ethereum / ethereum/execution-specs

Issue tracker to full benchmark test suite & glamsterdam-devnet-8 migration

Open
#3,375 0 comments 0 reactions 1 assignee Claimed by @LouisTsai-Csie View on GitHub
A-test-benchmark C-feat
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

## Summary

This tracker manages benchmark refactoring tasks to add `glamsterdam-devnet-8` support. Currently, benchmarks only work with `glamsterdam-devnet-7`.

- Payload Generation: Rafael generates via Benchmarkoor (not EELS)
- EELS Branch: `benchmarks/amsterdam`. Ensure new spec changes sync to `forks/amsterdam`
- Timeline: ~1 day for payload generation; Expect benchmarks start running on Friday.

## Scope

This includes refactoring benchmark cases for `glamsterdam-devnet-8` and adding new ones. We also need to refactor existing the infrastructure to support this.

### Benchmark Enhancement
For full suite benchmark, we need to add as much worst benchmark test as much as possible:
- [ ] Tracker: https://github.com/ethereum/execution-specs/issues/3329
- [ ] Open PRs: #3282

### Benchmark Refactor
In order to migrate to `glamsterdam-devnet-8` spec, the following cases needs refactoring:
- `tests/benchmark/stateful/eip7928_block_level_access_lists/` all failing: the top-frame delegation-resolution charge of `COLD_ACCOUNT_ACCESS` (3,000) is missing from `intrinsic_gas`.
- `test_block_full_access_list_and_data` is currently skipped if EIP-7934 block RLP size limit defined.
- Cold-storage benchmarks now need ~43% more setup, into blocks sized wrongly: please check `tx_iterations_by_gas_limit` returns roughly 43% more iterations for the same `gas_benchmark_value`, and every one of those slots has to be SSTORE-initialised in the setup phase.
- Value-bearing CALL benchmarks underfill, and devnet-8 widens the gap: The loop model charges the full `CALL_VALUE` per iteration, but the callee returns the unused 2,300 stipend, so the transaction consumes less than its limit and the block underfills its nominal target.
- Re-verify that the clear and oscillation cases still saturate the refund cap under the new constants, and that no further parametrisation has become degenerate. (`tests/benchmark/stateful/bloatnet/test_sstore.py:: test_sstore_dirty_transitions`)

- [ ] There were some failure in prev benchmark full test suite, they should be refactored before the next run: https://github.com/ethereum/execution-specs/issues/3281

### Framework Changes
The latest Jochemnet stateful benchmark is based on `bench-stateful-filler-optimizations`, we need to backport all the changes to `forks/amsterdam`, `benchmarks/amsterdam`.
- [ ] PR #3304 : instead of writing deployment script, embed this logic in the framework, or bloating the network to store these accounts.
- [ ] PR #3302
- [x] PR #3303
- [x] PR #3345
- [ ] PR #3348
- [x] PR #3367

Commits only in `bench-stateful-filler-optimizations` that needs to create PR, and merge into main/dev branch:
- Commits #6355360eb7, #65a7f54302, #9259eff68b, #bb2f19909d
- [ ] PR https://github.com/ethereum/execution-specs/pull/3379

### Cleanup Work
Some benchmark tests refactor, removal, code readability refactoring.
- [x] PR #3374 waiting review from CPerezz

### Additional Work

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.