Skip to content

Strip --minimum-expected-tests on retry attempts in RetryOrchestrator#8719

Merged
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/issue-5639-min-tests-with-retry
Jun 1, 2026
Merged

Strip --minimum-expected-tests on retry attempts in RetryOrchestrator#8719
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/issue-5639-min-tests-with-retry

Conversation

@Evangelink
Copy link
Copy Markdown
Member

Fixes #5639.

When --minimum-expected-tests was combined with --retry-failed-tests, the retry attempt (which only re-runs the previously failed tests) inherited the threshold computed against the full first-attempt test set. The retry's much smaller test count then tripped the policy and the run exited with code 9 (MinimumExpectedTestsPolicyViolation), even when the original first attempt satisfied the threshold.

This change makes RetryOrchestrator strip --minimum-expected-tests from the per-iteration argument list before launching any retry attempt (i.e. when lastListOfFailedId is non-empty). The first attempt still honors the option because it runs the full set of tests.

Adds an acceptance test RetryFailedTests_WithMinimumExpectedTests_StripsThresholdOnRetry that exercises the scenario from the issue across all target frameworks and asserts the run completes successfully without the policy violation message.

The retry orchestrator re-runs only the previously failed tests on retry
attempts. When --minimum-expected-tests was specified, the threshold
computed against the full test set was inherited by the retry attempt
and tripped the policy (exit code 9, MinimumExpectedTestsPolicyViolation),
even when the original first attempt satisfied it.

Drop --minimum-expected-tests from the per-iteration argument list whenever
we are about to issue a retry attempt (i.e. lastListOfFailedId is non-empty).
The first attempt still honors the option because it runs the full set.

Fixes #5639.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 31, 2026 18:54
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a bug where combining --minimum-expected-tests with --retry-failed-tests would cause retry attempts to fail with MinimumExpectedTestsPolicyViolation (exit code 9), since retries only re-run the failed subset and would never meet the original threshold.

Changes:

  • Strip --minimum-expected-tests from per-iteration arguments in RetryOrchestrator when launching retry attempts.
  • Add an acceptance test verifying the scenario across all target frameworks.
Show a summary per file
File Description
src/Platform/Microsoft.Testing.Extensions.Retry/RetryOrchestrator.cs Adds RemoveOption call for MinimumExpectedTestsOptionKey in the retry-attempt branch.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/RetryFailedTestsTests.cs New acceptance test RetryFailedTests_WithMinimumExpectedTests_StripsThresholdOnRetry.

Copilot's findings

  • Files reviewed: 2/2 changed files
  • Comments generated: 0

Copy link
Copy Markdown
Member Author

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ 21/21 dimensions clean — no findings.

Summary

The fix is minimal, correctly scoped, and addresses the root cause:

  • RemoveOption(finalArguments, PlatformCommandLineProvider.MinimumExpectedTestsOptionKey) is placed inside the if (lastListOfFailedId is { Length: > 0 }) guard, so the first attempt retains the threshold while every retry attempt—which re-runs only the previously failed subset—has it stripped. That's exactly the right boundary.
  • The acceptance test RetryFailedTests_WithMinimumExpectedTests_StripsThresholdOnRetry follows existing conventions (DynamicData over TargetFrameworks.AllForDynamicData, Guid.NewGuid() result directory, env-var-driven asset behavior) and verifies both the happy-path exit code and the absence of the violation message.
  • No public API surface was changed; no IPC wire format was altered; no new allocations on hot paths; threading model is unaffected.

Generated by Expert Code Review (on open) for issue #8719 · sonnet46 1.6M

@Evangelink Evangelink merged commit 7bc16cb into main Jun 1, 2026
33 of 34 checks passed
@Evangelink Evangelink deleted the dev/amauryleve/issue-5639-min-tests-with-retry branch June 1, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Revise --minimum-expected-tests behavior with Retry extension

2 participants