ethereum / ethereum/execution-specs
bug(tests-benchmark): Fix benchmark tests incompatible with fixed-opcode-count
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
> This should be resolved in a separate PR. The CI sometimes takes over 6 hours in fixed-opcode-count mode, which is unexpected. In normal CI pipeline, running 1K opcode count benchmarks completes in ~20 minutes, while the fixed-opcode-count config takes ~3 min.
>
> The root cause is in the collection/filtering logic. Tests without a code generator (i.e., those that provide `blocks=` or `tx=` directly) are only skipped _after_ the test body has already executed. The flow is:
>
> 1. The test body runs: pre-allocation, transaction construction, calldata generation, etc.
> 2. The test calls `benchmark_test(blocks=...)`, which instantiates `BenchmarkTest`.
> 3. `model_post_init` checks for a code generator, if absent, the test is skipped via `pytest.skip()`.
>
> The expensive work in step 1 is wasted. For uncacheable precompile benchmarks, which construct blocks manually rather than using the code generator, this is especially costly: in fixed-opcode-count mode, `gas_benchmark_value` is set to 1 trillion gas. With Osaka's 16M transaction gas cap, the transaction-building loop iterates ~60,000 times per test: generating calldata, funding EOAs, and creating `Transaction` objecte, all before the inevitable skip.
>
> The fix should move the skip earlier, either by filtering these tests at collection time in `pytest_collection_modifyitems`, or by skipping in the `gas_benchmark_value` fixture when it detects the test will not use a code generator.
>
_Originally posted by @LouisTsai-Csie in [#2600](https://github.com/ethereum/execution-specs/pull/2600/changes#r3062312277)_
Contributor guide
Assessment
This issue has not been assessed yet.