[FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 by RocMarshal · Pull Request #28003 · apache/flink

RocMarshal · 2026-04-23T04:09:30Z

What is the purpose of the change

[FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487

Brief change log

[FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487

Verifying this change

N.A

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

RocMarshal · 2026-04-23T04:11:55Z

Hi, @XComp @spuru9 @och5351 Could you help take a look ?
Any input is appreciated.

flinkbot · 2026-04-23T04:17:31Z

CI report:

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

spuru9

Thanks for the PR. Added a few comments.

nit: Other than these, there are a few trivial punctuation, fixing not relevant.

spuru9

Can you apply the changes for chinese doc as well.

spuru9

Sorry, Missed a minor inconsistency.

Rest LGTM

spuru9

LGTM, Thanks.

davidradl · 2026-04-23T11:58:05Z

+- `History`  
+  This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration. 
+  Additionally, the page supports the display of detailed rescale information as outlined below:
+    - The basic information of the target rescale


do we need to say target rescale- can we just say rescale or are there other rescale types?

davidradl · 2026-04-23T11:58:44Z

+    - The basic information of the target rescale
+        - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+        - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+        - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters


is this a UUID like the other one ?

yes.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-RescaleID&ResourceRequirementsrequest

davidradl · 2026-04-23T12:00:40Z

+        - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+        - Trigger Cause: The reason that triggered the target rescale
+        - Terminal State: The end state of the target rescale
+        - Terminated Reason: The reason for the completion or termination of the target rescale


it talks of Terminated Reason which implies a termination, then in the wording in talks of completions and terminations as 2 different things. Implying that termination is an abnormal end.

davidradl · 2026-04-23T12:57:17Z

+        - Terminated Reason: The reason for the completion or termination of the target rescale
+        - Start Time: The start time of the target rescale
+        - Duration: Duration from the start of the rescale to its completion or until now
+        - End Time: The end time of the target rescale


I assume this is the same as Duration unless the rescale is ongoing

consize.
And I had a try on a new description.

davidradl · 2026-04-23T12:57:37Z

+        - Start Time: The start time of the target rescale
+        - Duration: Duration from the start of the rescale to its completion or until now
+        - End Time: The end time of the target rescale
+    - The basic attributes and rescale change per `Job Vertex`


can we have a reference to Job vertex

Thanks @davidradl
I have considered this issue before. To be honest, JobVertex is such a fundamental term that I didn’t add any external reference links.

Of course , if you think it's required, I'd like to try to added it.

davidradl · 2026-04-23T12:58:01Z

+        - Duration: Duration from the start of the rescale to its completion or until now
+        - End Time: The end time of the target rescale
+    - The basic attributes and rescale change per `Job Vertex`
+        - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters


I assume JobVertx is the same as Job Vertex- I would suggest we are consistent.

Yes, +1 from me.

davidradl · 2026-04-23T12:59:21Z

+        - Acquired Parallelism: The parallelism of target vertex after the current rescale
+        - Sufficient Parallelism: The minimal parallelism of target vertex to run
+        - Desired Parallelism: The desired parallelism of the target vertex
+    - The basic attributes and rescale change per `Slot Sharing Group`


I would say UUID - and define that once

davidradl · 2026-04-23T13:00:38Z

+        - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+        - Request Profile: The request resource profile of the slot sharing group in the rescale
+        - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+    - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale


Are the states documented?

these states are an abstract representation of the scheduler's internal state; therefore, they were omitted to avoid introducing unnecessary ambiguity.

I missed this comment. @davidradl has a point here. We could link to the StateMachine section of FLIP-160. That's where it is documented.

davidradl · 2026-04-23T13:01:11Z

+        - State: The scheduler state name
+        - Enter Time: The time to enter the state
+        - Leave Time: The time to leave the state
+        - Duration: Time spent in the state (Leave Time − Enter Time)


does this include the current state which will not have an end time?

this doesn't include the current state which will not have an end time.

This is determined by the collection mechanism of adaptive state. Simply put, when a rescale adds a state, that state already includes start and end times.

davidradl · 2026-04-23T13:01:35Z

+        - Enter Time: The time to enter the state
+        - Leave Time: The time to leave the state
+        - Duration: Time spent in the state (Leave Time − Enter Time)
+        - Exception: The exception information about current rescale during the state


what does during the state mean?

maybe during-> within?

davidradl · 2026-04-23T13:04:01Z

+  This sub-page displays the total number of rescale events that have occurred since the job was launched,
+  along with the respective counts of failures and successes. 
+  Additionally, it provides statistical summaries of the rescale history, 
+  such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `Pnn` metrics.


what is Pnn here?

RocMarshal · 2026-04-27T02:35:12Z

This PR hasn’t received any new comments for four days. If this continues, I plan to merge it on Tuesday.
Thanks all involved in the review.

XComp

Sorry for the delay in responding. Looks good in general. Great job. 👍 I have a few cosmetic comments. PTAL

… FLIP-487 Co-authored-by: spuru9 <sinhapurushottam911@gmail.com> Co-authored-by: XComp <mpohl@confluent.io> Co-authored-by: davidradl <david_radley@uk.ibm.com>

RocMarshal · 2026-04-27T16:00:19Z

Thank you @XComp very much for your review.
Updated.
PTAL if you had the free time.
any input is appreciated 🌹

spuru9 suggested changes Apr 23, 2026

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Apr 23, 2026

spuru9 reviewed Apr 23, 2026

View reviewed changes

Comment thread docs/content/docs/deployment/elastic_scaling.md Outdated

RocMarshal force-pushed the FLINK-38902 branch 2 times, most recently from efce38d to 6d2a7cc Compare April 23, 2026 09:23

spuru9 reviewed Apr 23, 2026

View reviewed changes

Comment thread docs/content/docs/deployment/elastic_scaling.md Outdated

Comment thread docs/content.zh/docs/deployment/elastic_scaling.md Outdated

RocMarshal force-pushed the FLINK-38902 branch from 6d2a7cc to a5a7f40 Compare April 23, 2026 10:19

spuru9 approved these changes Apr 23, 2026

View reviewed changes

davidradl reviewed Apr 23, 2026

View reviewed changes

Comment thread docs/content.zh/docs/deployment/elastic_scaling.md Outdated

davidradl reviewed Apr 23, 2026

View reviewed changes

Comment thread docs/content.zh/docs/deployment/elastic_scaling.md Outdated

davidradl reviewed Apr 23, 2026

View reviewed changes

RocMarshal force-pushed the FLINK-38902 branch from a5a7f40 to e700ec6 Compare April 23, 2026 16:22

RocMarshal requested a review from davidradl April 24, 2026 10:25

XComp reviewed Apr 27, 2026

View reviewed changes

RocMarshal and others added 2 commits April 27, 2026 23:58

[FLINK-38902][docs] Add user instructions and usage documentation for…

3e09159

… FLIP-487 Co-authored-by: spuru9 <sinhapurushottam911@gmail.com> Co-authored-by: XComp <mpohl@confluent.io> Co-authored-by: davidradl <david_radley@uk.ibm.com>

address XComp's comments.

32fcbac

RocMarshal force-pushed the FLINK-38902 branch from e700ec6 to 32fcbac Compare April 27, 2026 15:58

RocMarshal requested a review from XComp April 27, 2026 16:04

XComp reviewed Apr 28, 2026

View reviewed changes

Comment thread docs/content.zh/docs/deployment/elastic_scaling.md Outdated

Comment thread docs/content.zh/docs/deployment/elastic_scaling.md Outdated

Updated based on review comments.

d97818f

RocMarshal requested a review from XComp April 28, 2026 12:51

Updated based on review comments.

5f376de

Conversation

RocMarshal commented Apr 23, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

RocMarshal commented Apr 23, 2026

Uh oh!

flinkbot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

spuru9 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spuru9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spuru9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

spuru9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RocMarshal Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

flinkbot commented Apr 23, 2026 •

edited

Loading

spuru9 left a comment •

edited

Loading

RocMarshal Apr 23, 2026 •

edited

Loading

XComp left a comment •

edited

Loading