ethereum / ethereum/execution-specs
feat(benchmark,fill): add JIT Compiler warm-up sequences as EEST tests
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
**This issue is still TBD; needs discussion.** Also to be considered: Do zkevms also require warm-up sequences to be performed before they get benchmarked? cc @jsign.
Can and should we express the warm-up sequences that Nethermind performs on required clients (Nethermind, Besu) prior to the actual execution of benchmark tests as EEST "tests"?
Doing this will be necessary if EEST + Hive becomes the infra used to actually perform the benchmarks.
Ideas:
- These could be added in `./tests/benchmark/test_warmup.py`.
- These tests could receive an appropriate marker.
- Ideally, these warm-up test engine-x fixtures would be generated for every pre-alloc group automatically (i.e., once for every `env` and fork config).
- `consume enginex` could get an additional flag `--benchmark` that would indicate it should run the warm-up tests first and then the actual benchmark tests (for every pre-alloc group).
Contributor guide
Assessment
This issue has not been assessed yet.