gh-146073: Add fitness/exit quality mechanism for JIT trace frontend by cocolato · Pull Request #148089 · python/cpython

cocolato · 2026-04-04T12:54:16Z

Issue: Improving trace quality by tracking "fitness" and "exit quality" #146073

cocolato · 2026-04-06T16:38:30Z

It appears that the current parameters do not yet guarantee runtime safety; I will continue to work on fixes and optimizations.

markshannon · 2026-04-07T08:28:12Z

I've commented on the issue #146073 (comment)

cocolato · 2026-04-11T14:03:24Z

@markshannon @Fidget-Spinner gentle ping, I’d like to hear your suggestions about the current parameters

markshannon

I've a few comments, mostly broad ideas and suggestions, rather than anything that needs fixing.

I think we should merge this soon. We can tweak the parameters later as we refine our ideas.

markshannon · 2026-04-13T16:37:57Z

+ * Higher = trace is more willing to stop here. */
+#define EXIT_QUALITY_CLOSE_LOOP      (FITNESS_INITIAL / 2)
+#define EXIT_QUALITY_ENTER_EXECUTOR  (FITNESS_INITIAL * 3 / 8)
+#define EXIT_QUALITY_DEFAULT         (FITNESS_INITIAL / 8)


I'd increase this to make sure that the fitness cannot drop from above EXIT_QUALITY_DEFAULT to below EXIT_QUALITY_SPECIALIZABLE in a single uop.

markshannon · 2026-04-13T16:38:00Z

+ * N_BACKWARD_SLACK more bytecodes before reaching EXIT_QUALITY_CLOSE_LOOP,
+ * based on AVG_SLOTS_PER_INSTRUCTION. */
+#define N_BACKWARD_SLACK           50
+#define FITNESS_BACKWARD_EDGE      (FITNESS_INITIAL - EXIT_QUALITY_CLOSE_LOOP \


A backwards edge isn't necessarily bad, although too many suggests a poor trace.

Maybe don't penalize the first backwards edge much, but penalize subsequent ones a lot.
What we really want is to avoid unrolling a loop that doesn't include the trace start.

It would be easier to reason about, if we used a simpler calculation for the value.

markshannon · 2026-04-13T16:41:05Z

+
+/* Backward edge penalty for JUMP_BACKWARD_NO_INTERRUPT (coroutines/yield-from).
+ * Smaller than FITNESS_BACKWARD_EDGE since these loops are very short. */
+#define FITNESS_BACKWARD_EDGE_COROUTINE  (FITNESS_BACKWARD_EDGE / 4)


It is not the length of the loop that matter. We want to include yields in generators in the traces of the loop that calls them, whether that is a yield from loop or a for loop doesn't much matter.

cocolato · 2026-04-14T08:18:15Z

@markshannon Thanks for review! I'm holding off on changing the fitness parameters for now, but I can run some benchmarks if you think it's necessary.

Fidget-Spinner · 2026-04-14T08:34:16Z

Still seeing a big slowdown in richards on https://github.com/colesbury/fastmark:

Main of this branch:

Benchmark                     Time      Useful Work
richards                      115.9 ms      (100%)
richards_super                103.5 ms      (100%)

This branch:

Benchmark                     Time      Useful Work
richards                      119.2 ms      (100%)
richards_super                118.5 ms      (100%)

I'm going to check if this is affecting the optimizer somehow.

…ER_EXECUTOR for RESUME

Treat back edges as an exit, not a penalty, this way they are more likely to end at a backedge instead of ending at random spots

markshannon · 2026-04-14T14:18:01Z

Increasing the max trace length is only going to help if the trace is stopping too early.
How big are the traces on main for richards?

Fidget-Spinner · 2026-04-14T14:33:31Z

I think we can safely reduce the max trace length, let me do that.

There were two problems with the older code:

Branch penalty was treated as before instruction count, when it should be treated as the sum over the expected trace length.
Treating non-closing JUMP_BACKWARD as penalty rather than exit quality seems to make it such that we stop traces at a certain offset after seeing a JUMP_BACKWARD. This isn't what we want. instead, we want to treat it as an exit that is worth stopping at (to increase the likelihood of linking to another trace).

New code has almost no slowdown on Richards, and a huge speedup on telco benchmark.

Benchmark                     Time      Useful Work
richards                      113.4 ms      (100%)
deltablue                     196.4 ms      (100%)
raytrace                      267.3 ms      (100%)
nbody                         150.6 ms      (100%)
go                            111.5 ms      (100%)
telco                        3517.8 ms      (100%)

Main:

Benchmark                     Time      Useful Work
richards                      111.6 ms      (100%)
deltablue                     192.2 ms      (100%)
raytrace                      270.4 ms      (100%)
nbody                         151.1 ms      (100%)
go                            112.6 ms      (100%)
telco                        3809.7 ms      (100%)

markshannon

A few more comments.

There are a few cases where we are still special casing some situations that fitness should handle and can be removed.

markshannon · 2026-04-15T08:00:51Z

+
+/* Exit quality thresholds: trace stops when fitness < exit_quality.
+ * Higher = trace is more willing to stop here. */
+#define EXIT_QUALITY_CLOSE_LOOP      (FITNESS_INITIAL)


Suggested change

#define EXIT_QUALITY_CLOSE_LOOP (FITNESS_INITIAL)

#define EXIT_QUALITY_CLOSE_LOOP (FITNESS_INITIAL - AVG_SLOTS_PER_INSTRUCTION*4)

FITNESS_INITIAL is too high a value for this, but not by much. We want to unroll tiny loops a bit and, more importantly, we don't want to special case the start instruction to avoid zero length traces.

markshannon · 2026-04-15T08:13:04Z

+ * N_BACKWARD_SLACK more bytecodes before reaching EXIT_QUALITY_CLOSE_LOOP,
+ * based on AVG_SLOTS_PER_INSTRUCTION. */
+#define N_BACKWARD_SLACK           50
+#define EXIT_QUALITY_BACKWARD_EDGE (EXIT_QUALITY_CLOSE_LOOP / 2 - N_BACKWARD_SLACK * AVG_SLOTS_PER_INSTRUCTION)


NOTE:

The problem here is that when tracing loops, we are treating the start of the loop as the closing point, but we want to stop at the end of the loop otherwise.
We probably need to make the back edge quality calculation a bit more complex.

if the jump is to the loop closing point: exit_quality = 0 (to ensure loop is closed)

otherwise: exit_quality = high ~(FITNESS - 10 * AVG_SLOTS_PER_INSTRUCTION)

(this can be fixed in a separate PR if would complicate this PR too much)

I prefer to do this in next PR.

markshannon · 2026-04-15T08:13:07Z

+
+/* Backward edge penalty for JUMP_BACKWARD_NO_INTERRUPT (coroutines/yield-from).
+ * Smaller than FITNESS_BACKWARD_EDGE since we want to trace through them. */
+#define EXIT_QUALITY_BACKWARD_EDGE_COROUTINE  (EXIT_QUALITY_BACKWARD_EDGE / 8)


Why are we treating these backward edges differently? They may be in smaller loops, but N_BACKWARD_SLACK already handles that.

I think this will help tracer through groutines short loops.

markshannon · 2026-04-15T08:14:36Z

+/* Heuristic backward-edge exit quality: leave room for about 1 unroll and
+ * N_BACKWARD_SLACK more bytecodes before reaching EXIT_QUALITY_CLOSE_LOOP,
+ * based on AVG_SLOTS_PER_INSTRUCTION. */
+#define N_BACKWARD_SLACK           50


Suggested change

#define N_BACKWARD_SLACK 50

#define N_BACKWARD_SLACK 10

50 seems way too high to me. What is the rationale for so high a number?

markshannon · 2026-04-15T08:21:47Z

+{
+    // We need to check for this, otherwise the first instruction (JUMP_BACKWARD usually)
+    // is mistakenly thought of as an exit.
+    if (uop_buffer_length((_PyJitUopBuffer *)&tracer->code_buffer) > CODE_SIZE_NO_PROGRESS) {


Suggested change

if (uop_buffer_length((_PyJitUopBuffer *)&tracer->code_buffer) > CODE_SIZE_NO_PROGRESS) {

It should be impossible for fitness < quality should be impossible for traces this short.

markshannon · 2026-04-15T08:23:05Z

+    // We need to check for this, otherwise the first instruction (JUMP_BACKWARD usually)
+    // is mistakenly thought of as an exit.
+    if (uop_buffer_length((_PyJitUopBuffer *)&tracer->code_buffer) > CODE_SIZE_NO_PROGRESS) {
+        if (target_instr == tracer->initial_state.start_instr ||


Suggested change

if (target_instr == tracer->initial_state.start_instr ||

if (target_instr == tracer->initial_state.close_loop_instr) {

For non-loop traces the start and loop close are the same. For loop traces (starting on a backwards edge) we don't want to stop on the back-edge, but at the top of the loop.

markshannon · 2026-04-15T08:27:28Z

+            target_instr == tracer->initial_state.close_loop_instr) {
+            return EXIT_QUALITY_CLOSE_LOOP;
+        }
+        else if (target_instr->op.code == ENTER_EXECUTOR && !_PyJit_EnterExecutorShouldStopTracing(opcode)) {


Suggested change

else if (target_instr->op.code == ENTER_EXECUTOR && !_PyJit_EnterExecutorShouldStopTracing(opcode)) {

else if (target_instr->op.code == ENTER_EXECUTOR) {

The fitness should handle this. If fitness is high, we will continue tracing. If it is getting lower, then we want to stop at the ENTER_EXECUTOR to join up with an existing trace.

No there's an exception to this: we don't want to treat ENTER_EXECUTORS caused by RESUME as a EXIT_QUALITY_ENTER_EXECUTOR, and instead treat them as a default one. Stopping at RESUME forms small, fragmented, loop traces, which I've previously documented in my RESUME tracing PR as I saw actual slowdowns from it.

This is what _PyJit_EnterExecutorShouldStopTracing says:

// Continue tracing (skip over the executor). If it's a RESUME // trace to form longer, more optimizeable traces. // We want to trace over RESUME traces. Otherwise, functions with lots of RESUME // end up with many fragmented traces which perform badly. // See for example, the richards benchmark in pyperformance. // For consideration: We may want to consider tracing over side traces // inserted into bytecode as well in the future.

I disagree.
The whole point of fitness/quality is remove these ad-hoc special cases.

The reason you were seeing many fragmented traces before was that we didn't have a principled way to do this.

We won't see lots of small fragmented traces because, as I said, the fitness will be high for short traces and will exceed EXIT_QUALITY_ENTER_EXECUTOR

I just did a benchmark. It's 0.5% slower applying this change on fastmark.

Can we introduce EXIT_QUALITY_ENTER_EXECUTOR_RESUME? We need to differentiate the following ENTER_EXECUTORS (they have different qualities in reality):

ENTER_EXECUTOR due to JUMP_BACKWARD (best).

ENTER_EXECUTOR due to progress or is_control_flow (decent)

ENTER_EXECUTOR due to RESUME (worst).

bedevere-app · 2026-04-15T08:32:18Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

cocolato and others added 13 commits April 1, 2026 00:57

add fitness && exit quality mechanism

1bfa176

Rewrite the code structure

2f9438a

address review

709c0a1

address many reviews

ef6ac24

Merge branch 'main' into jit-tracer-fitness

21f7122

optimize some constants

b99fe61

fix comment

d09afb5

fix constent

c9957c3

reduce frame penalty

9447546

add debug log

7d3e4c4

address review

2c1b5e0

address review

2409b2f

Merge branch 'python:main' into jit-tracer-fitness

88a91dc

cocolato requested review from FFY00, Fidget-Spinner, ZeroIntensity, ericsnowcurrently and markshannon as code owners April 4, 2026 12:54

bedevere-app bot added the awaiting review label Apr 4, 2026

bedevere-app bot mentioned this pull request Apr 4, 2026

Improving trace quality by tracking "fitness" and "exit quality" #146073

Open

cocolato added the skip news label Apr 4, 2026

cocolato added 2 commits April 6, 2026 13:51

Merge branch 'main' into jit-tracer-fitness

4e12f04

fine tune parameters

4bd251e

This comment was marked as outdated.

Sign in to view

remove some special cases

1d93208

cocolato added 3 commits April 10, 2026 17:50

Merge branch 'main' into jit-tracer-fitness

386c23a

rewrite fitness mechanism

83fd8ab

remove static assert

c900563

This comment was marked as outdated.

Sign in to view

Merge branch 'main' into jit-tracer-fitness

97d8be4

markshannon reviewed Apr 13, 2026

View reviewed changes

cocolato and others added 4 commits April 14, 2026 10:49

Merge branch 'main' into jit-tracer-fitness

559b164

address partial review

7a5e1fe

restore slots_rev

9324df0

address review

e69443b

Fidget-Spinner added 6 commits April 14, 2026 19:05

Race MAX_TARGET_LENGTH to 800, compute branch after slots, ignore ENT…

751a1d9

…ER_EXECUTOR for RESUME

reduce MAX_TARGET_LENGTH

896e4fe

fix tests

1364159

fix a bug

9fbec75

magic numbers

76b9c9e

Treat back edges as an exit, not a penalty, this way they are more likely to end at a backedge instead of ending at random spots

lint

d565f41

cocolato commented Apr 14, 2026

View reviewed changes

Comment thread Python/optimizer.c

This comment was marked as outdated.

Sign in to view

reduce the trace length to less than half

598d332

markshannon requested changes Apr 15, 2026

View reviewed changes

bedevere-app bot removed the awaiting review label Apr 15, 2026

bedevere-app bot added the awaiting changes label Apr 15, 2026

Fidget-Spinner added 2 commits April 15, 2026 17:48

Address review

64f3468

Reduce ENTER_EXECUTOR's exit quality

7661e7b

	#define EXIT_QUALITY_CLOSE_LOOP (FITNESS_INITIAL)
	#define EXIT_QUALITY_CLOSE_LOOP (FITNESS_INITIAL - AVG_SLOTS_PER_INSTRUCTION*4)

	if (target_instr == tracer->initial_state.start_instr \|\|
	if (target_instr == tracer->initial_state.close_loop_instr) {

	else if (target_instr->op.code == ENTER_EXECUTOR && !_PyJit_EnterExecutorShouldStopTracing(opcode)) {
	else if (target_instr->op.code == ENTER_EXECUTOR) {

Uh oh!

Conversation

cocolato commented Apr 4, 2026 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

cocolato commented Apr 6, 2026

Uh oh!

markshannon commented Apr 7, 2026

Uh oh!

This comment was marked as outdated.

cocolato commented Apr 11, 2026

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cocolato commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fidget-Spinner commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

markshannon commented Apr 14, 2026

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Fidget-Spinner commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fidget-Spinner Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-app bot commented Apr 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cocolato commented Apr 4, 2026 •

edited by bedevere-app bot

Loading

cocolato commented Apr 14, 2026 •

edited

Loading

Fidget-Spinner commented Apr 14, 2026 •

edited

Loading

Fidget-Spinner commented Apr 14, 2026 •

edited

Loading

Fidget-Spinner Apr 15, 2026 •

edited

Loading