ethereum / ethereum/execution-specs
Benchmark Full Suite Generation Tracker
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 106
Description
## Description
To finalize repricing numbers, we need to run the complete benchmark test suite on both compute and stateful benchmarks (not repricing-specific).
Initial results on Benchmarkoor:
- [state-actor-glamsterdam-devnet-7-full-stateful](https://benchmarkoor.core.ethpandaops.io/suites/0c5715e398c47aa3)
- [state-actor-glamsterdam-devnet-7-full-compute](https://benchmarkoor.core.ethpandaops.io/suites/9dbe0f5f4ea081b8)
This tracker documents what's missing or broken in the output.
## Compute
Statistics: passed 2040, failed 204, skipped 22, deselected 2.
These scenarios are not constructed successfully via the stateful filler. The constructed blocks are not accepted by clients:
- [x] `test_ether_transfers_to_precompile`
- [x] `test_ether_transfers`: diff_acc* ignored (66/120)
- [x] `test_auth_transaction`: `empty_authority_False` ignored (16/32)
- [x] `test_storage_access_cold`: `absent_slots_False` ignored (14/28)
- [x] `test_selfdestruct_existing`
- [x] `test_ext_account_query_cold`: `absent_accounts_False` (2/4)
- [x] `test_mixed_dependency_graph`: `all max_txs + half of half_max_txs (10/18)`
- [x] `test_deploy_then_interact`: `max_txs-single_contract` (2/12)
Refactor PR: issue #3282
Below tests needs further refactor, the current storage initialization approach would exceed the initcode prefix size limit.
- [ ] `test_state_root_computation`: initcode prefix too long
Below tests are being skipped:
- [ ] `test_unchunkified_bytecode`: contract deployment gas exceeds the benchmark gas value
- [ ] `test_creates_collisions`: Benchmark gas value cannot cover a single contract creation
- [ ] `test_bls12_pairing_uncachable[num_pairs_24]`: calldata intrinsic gas cost exceeds the gas limit, this could be refactored.
- [ ] `test_block_full_access_list_and_data`: EIP-7934 block RLP size limit might be exceeded -> this expected but worth refactoring
## Stateful
The test is limited to `tests/benchmark/stateful/bloatnet`, below is the failing cases:
1. `depth_benchmarks/test_deep_branch.py`
2. `eip7928_block_level_access_lists/`: not within the current scope.
3. `test_account_query.py`: `test_balance_query`, `test_extcodesize_bytecode_sizes`
4. `test_sload.py`: `test_sload_bloated_prefetch_miss`, `test_sload_bloated_multi_contract[existing_slots_True]`
5. `test_sstore.py`: `test_sstore_dirty_transitions[oscillation_4x_from_zero]`
6. `test_transaction_types.py`: `test_ether_transfers_onchain_receivers[to_self]`
7. `test_erc20.py`
8. `test_account_access`:non-existing scenario exceeds block gas limit.
9. `test_ether_transfers_onchain_receivers`: `diff_to_nonexistent` is not the worst case version, `diff_to_self` is broken due to RPC issue.
Suggestion to the next step:
1. Exclude p1, p2 and `test_sload_bloated_multi_contract` from this run, as this benchmark might require additional pre-state.
2. Remove tests in p3 and p7, since they are mostly covered by existing scenarios.
Comment here for visibility: https://github.com/ethereum/execution-specs/issues/3106#issuecomment-4934900449, now waiting for CPerezz' response.
This means we could focus on the following scenaio only:
- [ ] `test_sload_bloated_prefetch_miss`
- [ ] `test_sstore_dirty_transitions[oscillation_4x_from_zero]`
- [ ] `test_ether_transfers_onchain_receivers[to_self]`: *to_self costs only intrinsic gas per tx (12,000/tx), so a 300M gas target produces 25,000 transactions. build_block() RLP-encodes all of them into a single testing_buildBlockV1 request in one shot (it does not go through get_alloc's batching path) roughly 5.4-5.7MB total, which exceeds go-ethereum's plain HTTP RPC endpoint's hardcoded 5MB request-body limit (no CLI flag to raise it). At 60M the same case only produces 5,000 txs (~1.1MB), safely under the limit the 5x jump to 300M is what pushes it over.
- [x] `test_account_access` - #3282
- [x] `test_ether_transfers_onchain_receivers` - #3282
## Notes
- [x] Backports #3267 to `benchmarks/amsterdam` -> Done by PR #3276
- [x] Backports #3274 to `benchmarks/amsterdam` -> Done by PR #3277
Contributor guide
Research direction
Start with the full benchmark results and the listed tests under tests/benchmark/stateful/bloatnet, then review the compute scenarios and the referenced issue #3282. Re-run the in-scope compute and stateful benchmarks, investigate the named failures or skips, and document which scenarios produce accepted client results or require exclusion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100