ci(e2e): unify Linux + Windows UI tests into one parallel workflow#1027
Merged
Conversation
Adds three more vscode-java-dependency commands to the E2E plans, each
exercised through its documented user-visible UI surface:
* java-dep-new-types.yaml
- Adds a 7th cycle for newJavaRecord between Annotation and Abstract
Class. The Record option is filtered out for projects below Java 16
(see JavaType.getDisplayNames in src/explorerCommands/new.ts); the
fixture pom.xml is Java 17 so the option renders. Plan goes from
64 → 75 steps.
* java-dep-build-lifecycle.yaml
- Adds Test 6: java.project.update via the Maven submenu on the
project context menu (the command's only UI mount — no palette,
no toolbar). Uses the new 'contextMenuSubmenu' autotest action
that hovers a submenu trigger then clicks a leaf inside the
nested menu.
- Adds Test 7: java.project.clean.workspace via the view-title
overflow menu. The command may or may not raise a confirmation
dialog depending on JDT.LS state; we use the tolerant
tryClickDialogButton variant so the step passes either way
(cancelling the dialog if shown, avoiding the destructive
VS Code reload). Plan goes from 18 → 26 steps.
Requires @vscjava/vscode-autotest >= 0.7.11 (adds contextMenuSubmenu).
Locally verified:
java-dep-new-types.yaml 75/75 passing
java-dep-build-lifecycle.yaml 26/26 passing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Refactor linuxUI.yml and windowsUI.yml from a single sequential job
running autotest run-all into a 5-job parallel pipeline modelled after
vscode-java-pack/.github/workflows/e2e-autotest.yml:
1. build-<os>: produce VSIX once, upload as artifact
2. discover-plans: scan test/e2e-plans/*.yaml -> matrix output
3. e2e-<os> (matrix per plan): download VSIX, run a single plan
4. analyze-<os>: aggregate results.json across plans into summary.md
5. linux-ui / windows-ui: gate job preserving the original required
status-check name so existing branch protection keeps working
Per-job timeout drops to 25 min and individual plan failures surface
independently. Sub-screenshots from each plan are preserved per-plan
in their own artifact for easier triage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Merges the per-OS workflows (linuxUI.yml, windowsUI.yml) into a single
e2eUI.yml with a unified entry point and a single shared summary.
Structure:
- discover-plans (matrix output)
- build-linux / build-windows (VSIX artifacts)
- e2e-linux / e2e-windows (matrix) (one job per (OS, plan))
- analyze (one combined summary across both OSes,
plan dirs prefixed by OS for clarity)
- Linux-UI / Windows-UI (gate jobs) (preserve required status-check names)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
wenytang-ms
changed the title
Reworks the e2e workflow so {os, plan} is a single cartesian matrix
instead of two parallel jobs (e2e-linux, e2e-windows):
- lint tslint + checkstyle (ubuntu, OS-agnostic)
- discover-plans
- build (matrix on os)
- e2e (matrix on {os, plan}) cartesian 2 x N parallel jobs
- analyze one combined summary across both OSes
- Linux-UI / Windows-UI gates preserve required-check names
Inspired by vscode-java-pack/e2e-autotest.yml extended with an OS
dimension. OS-specific steps (Xvfb, shell flavor) are guarded by
runner.os conditionals.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Each matrix cell already surfaces as its own PR check, so a separate gate is unnecessary. Branch protection should be updated to require the individual matrix checks (or the 'E2E Summary' job) directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the repository’s E2E UI CI by replacing the separate Linux/Windows workflows with a single unified workflow that builds VSIX artifacts per OS and runs UI plans in parallel via a matrix, then aggregates results into one summary. It also updates two E2E plan YAMLs to expand UI coverage (notably adding a “Record” creation flow and adding project “Reload/Clean” coverage).
Changes:
- Replace
.github/workflows/linuxUI.yml+windowsUI.ymlwith a consolidated.github/workflows/e2eUI.ymlusing matrix parallelism and a unified analysis job. - Extend
java-dep-new-types.yamlto include “New Record” coverage and renumber subsequent tests. - Extend
java-dep-build-lifecycle.yamlto coverjava.project.update(via Maven submenu) andjava.project.clean.workspace(via view overflow).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
test/e2e-plans/java-dep-new-types.yaml |
Adds “New Record” flow and shifts later steps to keep coverage of all “New …” options. |
test/e2e-plans/java-dep-build-lifecycle.yaml |
Adds additional build-lifecycle command coverage (update + clean workspace) using UI entry points. |
.github/workflows/windowsUI.yml |
Removes the legacy Windows-only UI workflow. |
.github/workflows/linuxUI.yml |
Removes the legacy Linux-only UI workflow. |
.github/workflows/e2eUI.yml |
Introduces a unified, parallel OS×plan E2E UI workflow with shared analysis/summary. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Folding OS into the e2e matrix forced every cell to wait for the entire build matrix to finish (Linux + Windows), so a slow Windows build delayed Linux e2e jobs by ~2.5 min unnecessarily (Copilot review feedback). Split back into independent per-OS pipelines: build-linux -> e2e-linux (matrix on plan) build-windows -> e2e-windows (matrix on plan) Both pipelines feed into the shared discover-plans + analyze jobs, so the unified entry point and combined summary are preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
chagong
approved these changes
Jun 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates the two former UI workflow files (
linuxUI.yml,windowsUI.yml) into a singlee2eUI.ymlwith per-OS pipelines and per-plan parallelism via matrix strategy.Modelled after
vscode-java-pack/.github/workflows/e2e-autotest.yml.Background: with 5+ e2e plans (and growing) the previous single sequential job (
autotest run-all) was pushing past the 30-minute per-job timeout. Splitting into per-plan matrix jobs lets every plan run independently.New pipeline (single workflow file)
lintdiscover-planstest/e2e-plans/*.yaml→ emit JSON matrixbuild-linux/build-windowsvsce package→ upload per-OS VSIX artifacte2e-linux(matrix per plan)build-linux. Downloads VSIX, runs plan under Xvfb 1920×1080e2e-windows(matrix per plan)build-windows. Downloads VSIX, runs plananalyzeif: always()— downloads all plan results from both OSes, prefixes each plan dir by OS (linux-…,windows-…), runsautotest analyzeonce, writes one combinedsummary.mdinto$GITHUB_STEP_SUMMARYNotes
fail-fast: false— a single flaky plan doesn't cancel its siblings.Linux-UI/Windows-UIchecks will need to be updated to require eitherE2E Summary(the analyze job) or the individual matrix check names (e.g.E2E Linux (java-dep-build-lifecycle)), otherwise PRs will appear to wait forever for the old check names.e2e-results-<os>-<plan>), making per-plan triage easier.Follow-up
Independent of the Phase-A coverage PR; this PR only changes CI plumbing.