ethereum / ethereum/execution-specs

Tracking issue: fork/spec agnostic benchmarks

Open
#3,252 0 comments 0 reactions 1 assignee Claimed by @jochem-brouwer View on GitHub
A-test-benchmark C-feat C-refactor E-hard P-high
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

The generalized goal for benchmarks is to describe a scenario to stress-test the
client for. Example scenarios are: run as many `KECCAK256` operations, load as
many distinct maximum-sized contracts as possible, maximize the memory, spam as
much log receipt data as possible, change as many Merkle Patricia Tree
(intermediate) nodes as possible, or perform the most JUMPDEST analysis. These benchmarks target a specific goal X (usually: execute close to or equal to block gas limit `X` [^gas-dimensions], or execute (at least) `Y` times a specific opcode)

This is (usually) not dependent on the specs. We can probe a target client: this does not
have to be our EELS implementation; it is actually more useful to use the Client
Under Test (CUT).[^test-vs-benchmark] For instance, we do not need to know specific details, for instance exact gas costs of an opcode. If we know specific high level constraints (like tx gas limit (EIP-7825)) then we can probe the client to infer more information, and to get to the target.

## What this unlocks

Especially at the "frontier" of devnet specifications, details often change. For instance, a specific gas cost or an edge case. For generic benchmarks, we usually do not hit these specific edge case (of course, the full benchmark suite should do so), but these edge cases complicate the development. If we release a benchmark suite at the frontier, and the spec changes slightly, it could mean that at most the entire suite is invalid because the client runs in consensus issues.

**We therefore turn this idea around**

The benchmark test filler describes a situation of *what* to test. This is executed against the client. It is our (execution-specs benchmark (stateful) fill tool) to figure out *how* to test this and to reach the targets. For instance, if we have a code pattern to test for a specific situation (run this opcode X times, or edit state), we can probe the client on the behavior. If we target 30M gas to spend, we can send one transaction at the EIP-7825 maximum gas limit to see what happens (the 16.7 MGas should fit in it). If we then attempt to build a block with two of these, we likely run out of gas. We can use binary search to figure out the optimal situation, and guarantee using at least a lower bound of gas for the test to fill (this prevents benchmarks being filled but not executing what we target).
This means that on a spec change, a client can directly implement it, fill the test against itself, and then use that to optimize the performance. This means that **even a non-compliant client to specs can optimize their performance** and to fix the consensus problems at a later step. This should significantly speed up the performance/benchmark feedback cycle.

## Implementation

Benchmark files (`tests/benchamark`) describe the scenario to test. Our tooling guides the fill tool to determine the right inputs to the state transition function (STF) to generate the "correct" payloads. This means clients are not anymore dependent on us for their optimizations, they can do both the optimizations and the consensus in parallel, and if the consensus is updated they can directly optimize their own client in case there is a divergence in the payload status.

## Steps

- [ ] Identify the constraints/targets clients fill against (`X` opcode runs, `Y` gas limit, etc.)
- [ ] For each benchmark, determine if behavior is semi-hardcoded or not. The filler helpers must be converted to tools which probe (call multiple `testing_buildBlockV1`) the client behavior in order to reach the target. This target must be within specific limits (e.g. at most 5% less than block gas limit used)
- [ ] Write generalized test probes to help with this
- [ ] Semi-hardcoded benchmarks must be converted to this fork/spec agnostic ones

This indirectly also helps for the repricing suites, as this helps clients optimize their current branch, or any branch they are working on (fill against self using the tooling, and then visualize it on benchmarkoor)

(Note: text is written without LLM, the formatting has been helped with by Claude)

[^gas-dimensions]: The splitting of gas dimensions into state gas and execution
gas due to EIP-8037 (and EIP-2780, EIP-8038) as of Glamsterdam complicates
this somewhat.

[^test-vs-benchmark]: "Test" and "benchmark" are used interchangeably in this
issue. Both use the same framework; only the goals differ. The *test* is
used for consensus tests for clients, the *benchmark* for **performance**
tests. Both are important, and with the introduction of Block Access Lists
(BALs) in Glamsterdam performance is also part of the protocol: the
expectation that a client meets a specific performance level is now
implicitly part of the protocol specification.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.