ethereum / ethereum/execution-specs

Benchmark Full Suite Generation Tracker

Open
#3,281 1 comment 0 reactions 0 assignees View on GitHub
A-test-benchmark C-tracker
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 8h
Merged PRs (30d)
106

Description

## Description

To finalize repricing numbers, we need to run the complete benchmark test suite on both compute and stateful benchmarks (not repricing-specific).

Initial results on Benchmarkoor:
- [state-actor-glamsterdam-devnet-7-full-stateful](https://benchmarkoor.core.ethpandaops.io/suites/0c5715e398c47aa3)
- [state-actor-glamsterdam-devnet-7-full-compute](https://benchmarkoor.core.ethpandaops.io/suites/9dbe0f5f4ea081b8)

This tracker documents what's missing or broken in the output.

## Compute

Statistics: passed 2040, failed 204, skipped 22, deselected 2.

These scenarios are not constructed successfully via the stateful filler. The constructed blocks are not accepted by clients:
- [x] `test_ether_transfers_to_precompile`
- [x] `test_ether_transfers`: diff_acc* ignored (66/120)
- [x] `test_auth_transaction`: `empty_authority_False` ignored (16/32)
- [x] `test_storage_access_cold`: `absent_slots_False` ignored (14/28)
- [x] `test_selfdestruct_existing`
- [x] `test_ext_account_query_cold`: `absent_accounts_False` (2/4)
- [x] `test_mixed_dependency_graph`: `all max_txs + half of half_max_txs (10/18)`
- [x] `test_deploy_then_interact`: `max_txs-single_contract` (2/12)

Refactor PR: issue #3282

Below tests needs further refactor, the current storage initialization approach would exceed the initcode prefix size limit.

- [ ] `test_state_root_computation`: initcode prefix too long

Below tests are being skipped:
- [ ] `test_unchunkified_bytecode`: contract deployment gas exceeds the benchmark gas value
- [ ] `test_creates_collisions`: Benchmark gas value cannot cover a single contract creation
- [ ] `test_bls12_pairing_uncachable[num_pairs_24]`: calldata intrinsic gas cost exceeds the gas limit, this could be refactored.
- [ ] `test_block_full_access_list_and_data`: EIP-7934 block RLP size limit might be exceeded -> this expected but worth refactoring

## Stateful

The test is limited to `tests/benchmark/stateful/bloatnet`, below is the failing cases:

1. `depth_benchmarks/test_deep_branch.py`
2. `eip7928_block_level_access_lists/`: not within the current scope.
3. `test_account_query.py`: `test_balance_query`, `test_extcodesize_bytecode_sizes`
4. `test_sload.py`: `test_sload_bloated_prefetch_miss`, `test_sload_bloated_multi_contract[existing_slots_True]`
5. `test_sstore.py`: `test_sstore_dirty_transitions[oscillation_4x_from_zero]`
6. `test_transaction_types.py`: `test_ether_transfers_onchain_receivers[to_self]`
7. `test_erc20.py`
8. `test_account_access`:non-existing scenario exceeds block gas limit.
9. `test_ether_transfers_onchain_receivers`: `diff_to_nonexistent` is not the worst case version, `diff_to_self` is broken due to RPC issue.

Suggestion to the next step:

1. Exclude p1, p2 and `test_sload_bloated_multi_contract` from this run, as this benchmark might require additional pre-state.
2. Remove tests in p3 and p7, since they are mostly covered by existing scenarios.

Comment here for visibility: https://github.com/ethereum/execution-specs/issues/3106#issuecomment-4934900449, now waiting for CPerezz' response.

This means we could focus on the following scenaio only:
- [ ] `test_sload_bloated_prefetch_miss`
- [ ] `test_sstore_dirty_transitions[oscillation_4x_from_zero]`
- [ ] `test_ether_transfers_onchain_receivers[to_self]`: *to_self costs only intrinsic gas per tx (12,000/tx), so a 300M gas target produces 25,000 transactions. build_block() RLP-encodes all of them into a single testing_buildBlockV1 request in one shot (it does not go through get_alloc's batching path) roughly 5.4-5.7MB total, which exceeds go-ethereum's plain HTTP RPC endpoint's hardcoded 5MB request-body limit (no CLI flag to raise it). At 60M the same case only produces 5,000 txs (~1.1MB), safely under the limit the 5x jump to 300M is what pushes it over.
- [x] `test_account_access` - #3282
- [x] `test_ether_transfers_onchain_receivers` - #3282

## Notes
- [x] Backports #3267 to `benchmarks/amsterdam` -> Done by PR #3276
- [x] Backports #3274 to `benchmarks/amsterdam` -> Done by PR #3277

Contributor guide

Open the contributing guide

Research direction

Start with the full benchmark results and the listed tests under tests/benchmark/stateful/bloatnet, then review the compute scenarios and the referenced issue #3282. Re-run the in-scope compute and stateful benchmarks, investigate the named failures or skips, and document which scenarios produce accepted client results or require exclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.