ethereum / ethereum/execution-specs

Ensure benchmarks measure the intended access patterns

Open
#2,961 4 comments 0 reactions 1 assignee Claimed by @LouisTsai-Csie View on GitHub
A-test-benchmark C-feat stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

## Context

To ensure benchmark data correctness, we need to verify the benchmark implementation and on-chain stub layout match expectations. Several properties are especially necessary to verify, including warm/cold access patterns, account access patterns and storage slot access patterns.

We could add post-storage verification in consensus tests, but it's too expensive for benchmarking, as the overhead would skew the final results. Gas usage verification isn't feasible here either. For complex benchmarks, calculating exact gas costs during execution becomes too complicated.

## Properties
### Warm/Cold account access
For some benchmarks we need to guarantee that every account access in the benchmark loop is cold (or warm). For example, in `test_ext_account_query_warm` / `test_ext_account_query_cold`, which targets `BALANCE`, `EXTCODE*` and `CALL*`.

### Storage slot access
- `test_sload_bloated`: parametrized by `existing_slots`. When `existing_slots=True` the slots are pre-populated, so each `SLOAD` must hit an existing slot;
- `test_sstore_bloated`: parametrized by `existing_slots` and `write_new_value`. `existing_slots` controls whether the target slot is pre-populated (non-zero original) or empty; `write_new_value` controls whether each `SSTORE` writes a **new** value (original + 1) or rewrites the same value. These select different gas paths, so we must confirm he target slot is actually in the assumed state.

### Account access pattern
For `test_account_access`, the `account_mode` parametrization covers `NON_EXISTING_ACCOUNT`, `EXISTING_EOA`, and `EXISTING_CONTRACT`. How do we ensure the on-chain target actually matches the expected mode on the network?

## Proposed solution
### Gas-cost based (warm/cold + storage slot)
For warm/cold account access and storage slot access, the access cost itself
distinguishes the cases (EIP-2929):

- warm vs cold account access: `100` vs `2600` gas
- warm vs cold storage slot access: `100` vs `2100` gas

We can read the gas cost from the trace and check it matches expectation. There are some ongoing exploration works for this, including PR #2640 and @gurukamath 's [PoC](https://github.com/gurukamath/execution-specs/commit/531a033e593ee5ba6ec576231c6e2d775c979b22).

Open challenges:
- Generating the entire trace might be too large.
- The trace format is not unified across clients.
- given the stateful filling tooling, where does the trace come from? `debug_traceTransaction` or `t8n`

### Account access pattern
The account access pattern is invisible to gas cost: `EXISTING_CONTRACT` and `EXISTING_EOA` cost the same for a plain access.

Proposed solutions:

- Solution 1: valued `CALL` + revert (Jochem's approach)

Send a valued `CALL` to the target inside a sub-frame, then revert the frame. If the account did not exist, the trace shows the ` + 25000` new-account charge on that `CALL`; the revert rolls back the creation so the pre-state is preserved. Distinguishes exist vs non-exist only, and mutates/warms state (hence the revert).

- Solution 2: `EXTCODEHASH` self-asserting verification tx

Add a dedicated verification transaction that runs before the benchmark transaction. It probes each target with `EXTCODEHASH`, compares against the expected hash, and `REVERT`s (or hits `INVALID`) on mismatch. The receipt status then tells us pass/fail. No trace parsing required, and the check is enforced by the EVM itself. `EXTCODEHASH` resolves all three classes:

1. `0x0` → non-existing / empty account
2. `keccak256("")` (`c5d2460186...85a470`) → existing EOA (no code)
3. `keccak256(code)` → existing contract

Because it is a pure read it does not mutate state (no revert frame needed), but it does warm the target, so it must run in a separate transaction from the benchmark.

- Solution 3: RPC pre-flight query
The stateful filler already assumes an RPC endpoint. Before running the benchmark, query the target's true state and assert outside the EVM entirely:

1. `eth_getCode` → EOA (`0x`) vs contract vs 7702 (`0xef0100...`)
2. `eth_getBalance` / `eth_getTransactionCount` → account is non-empty
3. `eth_getStorageAt` → slot holds the expected value

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.