Gas repricing tooling requirements and coordination tracker

# Gas repricing benchmarking: tooling requirements and coordination

## Summary

We need a clearer communication method between the test fixtures and the benchmarking tooling so that gas repricing benchmarks can accurately control client cache behaviour (cached vs non-cached) and other execution details. Right now, tests are not able to express these requirements to the tooling in a structured way, which is blocking or slowing down data collection for gas repricing work.

This issue is to:

- Agree on how tests communicate execution requirements to tooling
- Define what the tooling must support for gas repricing benchmarks
- Decide on a minimal design we can implement quickly for `execute` (and ideally reuse for `fill` later)

---

## Problems today

1. **No explicit cache control from tests to tooling**
- For gas repricing we care about:
- cached vs non-cached account access
- cached vs non-cached storage access (SLOAD/SSTORE)
- Current tests only implicitly affect cache state (via setup phases, access lists, etc.), and tooling can’t reliably distinguish:
- tests that require “true cold / non-cached” access
- tests that intentionally rely on cached/warm data

2. **Tooling behaviour not driven by fixture metadata**
- Our tests already carry structured metadata internally (e.g. stage information, request IDs), but:
- We don’t expose enough of this metadata in a way that external tooling (e.g. Benchmarker, execute harnesses) can consume
- We have no standard place to express extra requirements (like “clear cache before this test”)

3. **Client cache behaviour and checkpointing**
- @skylenet ’s checkpoint approach can provide full isolation, but:
- Tools need to know _when_ to restore checkpoints or clear caches
- Some tests should run with default client caching, others explicitly without
- Without a signalling mechanism from the tests, tooling has to guess or apply a one-size-fits-all strategy.

4. **Tight timelines for gas repricing data**
- Gas repricing efforts needs benchmark data for gas repricing EIPs, particularly:
- Account access opcodes (BALANCE, EXTCODESIZE, EXTCODEHASH, EXTCOPY, calls)
- Storage ops (SLOAD, SSTORE) under different state sizes and cache conditions
- We are blocked or slowed down because the tests and tooling can’t coordinate cache semantics properly.

---

## Proposed direction

### 1. Define a minimal “execution requirements” schema

Introduce an explicit, minimal schema that tests can use to express what they need from the tooling. Examples of possible fields:

- `cache_mode`:
- `default` (tooling free to use normal client caching)
- `non_cached` (tooling must ensure no relevant data is cached before execution)
- `cached` (tooling should intentionally warm/cache data before execution, if not already)
- `state_size_profile` (optional hint, e.g. `small`, `medium`, `large`, or a named profile)
- `execution_stage`:
- Already implicitly present (e.g. setup vs execution), but we may need it exposed where tooling can read it.

We should keep this schema as small and generic as possible so it can be reused across forks and future benchmarks.

### 2. Expose test metadata in a tooling-friendly way

We already embed information such as test name, stage, etc. in:

- Request IDs per transaction
- Index files / fixture metadata

Action items:

- Ensure markers/requirements (e.g. `cache_mode`) are:
- Attached at the test case level (e.g. via pytest markers or equivalent)
- Serialized into a place the tooling can read:
- For `execute`: within the request ID or sidecar metadata, or
- In an index/manifest file that the tooling can consume

The key is: **tooling should not need to guess; it should read requirements from a single, well-defined location.**

### 3. Tooling behaviour changes (especially for `execute`)

For `execute`-based benchmarking (Amsterdam / gas repricing):

- When `cache_mode = non_cached`:
- Tooling must ensure the client is in a state where:
- Relevant accounts/slots have not been accessed in the current process lifetime
- Or, equivalently, restore a checkpoint that guarantees “cold” access
- @skylenet ’s checkpoint mechanism is a good fit here.

- When `cache_mode = cached`:
- Tooling can:
- Run a pre-execution warmup (e.g. a setup transaction or read) before the measured transaction(s), or
- Rely on a dedicated “setup” stage encoded in the test, if we provide it

- When `cache_mode = default`:
- Tooling can use its current behaviour (no special treatment beyond what it already does).

We should design this so that:

- Tools like Benchmarker can opt-in to honour these requirements
- Other consumers are not forced to implement everything on day one

### 4. Apply first to gas repricing benchmarks

Initial scope:

- Tests used for:
- Account access repricing
- Storage access repricing
- These tests will:
- Be explicitly marked with the appropriate `cache_mode`
- Use existing bloated contracts / mainnet-derived contracts (as per Louis/Carlos work)
- Tooling:
- Benchmarker / execute harness must implement the minimal behaviour for these tests
- Once it works for gas repricing, we can generalize to other benchmarks

---

## Open questions for discussion

1. **Exact encoding / transport of metadata**
- Should we rely on:
- Request IDs only?
- A sidecar index/manifest file (JSON/YAML) per fixture set?
- Both, depending on consumer?

2. **Granularity**
- Should `cache_mode` be per-test, per-transaction, or per-stage?
- For gas repricing benchmarks, is per-test enough, or do we need more fine-grained control?

3. **Interaction with checkpointing**
- How do we express “restore checkpoint before this test” vs “no checkpoint, run in continuous mode”?
- Should this be encoded as part of the same schema or handled out-of-band by individual tools?

4. **Backwards compatibility**
- How do we introduce this without breaking existing tools?
- Do we require tools to treat unknown / missing metadata as `cache_mode = default`?

5. **Execute vs fill**
- For gas repricing we care primarily about `execute` on devnets / mainnet-like environments.
- Do we want the same metadata model to be used for `fill` fixtures (e.g. for future stateful tests), or keep them separate?

---

## Next steps

- Align on the minimal schema (especially `cache_mode`) and where it should live
- Testing team:
- Prototype markers / metadata on the relevant benchmark tests
- Ensure metadata is serialized somewhere tooling can access
- Tooling (@skylenet, @kamilchodola , etc.):
- Specify how checkpointing / cache clearing will interpret the metadata
- Implement basic support for the gas repricing benchmark set
- Revisit after initial integration to:
- Adjust schema if needed
- Decide whether to extend this pattern to other test categories