Skip to content

feat: add benchmark baseline artifact #3277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tac0turtle wants to merge 1 commit into from
+468 −3
Closed

feat: add benchmark baseline artifact #3277

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,15 @@ jobs:
run: just build-evm build-da
- name: Run ERC20 throughput test
run: |
cd test/e2e && go test -tags evm \
cd test/e2e && BENCH_RESULT_OUTPUT=benchmark/erc20_run_result.json \
BENCH_OBJECTIVE='ERC-20 token transfer throughput in isolation' go test -tags evm \
-run='^TestSpamoorSuite$/^TestERC20Throughput$' -v -timeout=15m \
./benchmark/ --evm-binary=../../../build/evm
- name: Upload benchmark results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: erc20-benchmark-results
path: test/e2e/benchmark/erc20_run_result.json
Comment on lines +103 to +107
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the benchmark artifact upload steps fail when expected JSON outputs are absent.
rg -n -C 4 'name: (erc20|defi|state-pressure)-benchmark-results|if-no-files-found' .github/workflows/benchmark.yml

Repository: evstack/ev-node

Length of output: 1339

🌐 Web query:

actions/upload-artifact v7.0.1 if-no-files-found parameter documentation

💡 Result:

The if-no-files-found parameter for actions/upload-artifact v7.0.1 controls the behavior when no files match the provided path glob. Description from action.yml: "The desired behavior if no files are found using the provided path. Available Options: warn: Output a warning but do not fail the action, error: Fail the action with an error message, ignore: Do not output any warnings or errors, the action does not fail" Default: 'warn' Example usage: - uses: actions/upload-artifact@v7 with: name: my-artifact path: path/to/artifact/ if-no-files-found: error This parameter has remained unchanged since its introduction (around v2), through v7.0.0 and the patch release v7.0.1 (which only updated README examples and dependencies, no action.yml changes).

Citations:

Add if-no-files-found: error to make benchmark artifact uploads fail when expected JSON files are missing.

Lines 103-107, 131-135, and 160-164 upload benchmark JSON artifacts, but missing files currently won't fail the workflow. The actions/upload-artifact action defaults to warn behavior when files are missing. Since emitRunResult only logs write failures, the benchmarks can pass without producing the required artifacts. Add if-no-files-found: error to all three upload-artifact steps to enforce the artifact contract.

Proposed workflow hardening 
       - name: Upload benchmark results
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: erc20-benchmark-results
           path: test/e2e/benchmark/erc20_run_result.json
+          if-no-files-found: error
@@
       - name: Upload benchmark results
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: defi-benchmark-results
           path: test/e2e/benchmark/defi_run_result.json
+          if-no-files-found: error
@@
       - name: Upload benchmark results
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: state-pressure-benchmark-results
           path: test/e2e/benchmark/state_pressure_run_result.json
+          if-no-files-found: error
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/benchmark.yml around lines 103 - 107, The upload-artifact
steps that send benchmark JSONs currently allow missing files and should instead
fail the workflow; for each actions/upload-artifact step that uploads benchmark
results (the "Upload benchmark results" steps that set name:
erc20-benchmark-results and the two analogous benchmark artifact uploads), add
the input if-no-files-found: error to the step's with: block so the action
errors when the expected JSON (e.g., test/e2e/benchmark/erc20_run_result.json)
is missing.


defi-benchmark:
name: DeFi Simulation Benchmark
Expand All @@ -118,9 +124,15 @@ jobs:
run: just build-evm build-da
- name: Run DeFi simulation test
run: |
cd test/e2e && go test -tags evm \
cd test/e2e && BENCH_RESULT_OUTPUT=benchmark/defi_run_result.json \
BENCH_OBJECTIVE='Uniswap V2 swap throughput under a DeFi workload' go test -tags evm \
-run='^TestSpamoorSuite$/^TestDeFiSimulation$' -v -timeout=15m \
./benchmark/ --evm-binary=../../../build/evm
- name: Upload benchmark results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: defi-benchmark-results
path: test/e2e/benchmark/defi_run_result.json

# TODO: wire up to publish results once additional tests are in place.
state-pressure-benchmark:
Expand All @@ -141,9 +153,15 @@ jobs:
run: just build-evm build-da
- name: Run state pressure test
run: |
cd test/e2e && go test -tags evm \
cd test/e2e && BENCH_RESULT_OUTPUT=benchmark/state_pressure_run_result.json \
BENCH_OBJECTIVE='Storage-heavy throughput under maximum state growth pressure' go test -tags evm \
-run='^TestSpamoorSuite$/^TestStatePressure$' -v -timeout=15m \
./benchmark/ --evm-binary=../../../build/evm
- name: Upload benchmark results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: state-pressure-benchmark-results
path: test/e2e/benchmark/state_pressure_run_result.json

# single job to push all results to gh-pages sequentially, avoiding race conditions
publish-benchmarks:
Expand Down
Loading
Loading