ethereum / ethereum/execution-specs

feat(benchmark,fill): add JIT Compiler warm-up sequences as EEST tests

Open
#1,561 5 comments 0 reactions 0 assignees View on GitHub
A-test-benchmark A-test-fill S-needs-discussion stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

**This issue is still TBD; needs discussion.** Also to be considered: Do zkevms also require warm-up sequences to be performed before they get benchmarked? cc @jsign.

Can and should we express the warm-up sequences that Nethermind performs on required clients (Nethermind, Besu) prior to the actual execution of benchmark tests as EEST "tests"?

Doing this will be necessary if EEST + Hive becomes the infra used to actually perform the benchmarks.

Ideas:
- These could be added in `./tests/benchmark/test_warmup.py`.
- These tests could receive an appropriate marker.
- Ideally, these warm-up test engine-x fixtures would be generated for every pre-alloc group automatically (i.e., once for every `env` and fork config).
- `consume enginex` could get an additional flag `--benchmark` that would indicate it should run the warm-up tests first and then the actual benchmark tests (for every pre-alloc group).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.