ethereum / ethereum/execution-specs

Enable t8n caching for state_test on Paris/Shanghai

Open
#2,225 1 comment 0 reactions 0 assignees View on GitHub
A-test-fill C-feat stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

## Background

[PR #2084](https://github.com/ethereum/execution-specs/pull/2084) introduced t8n output caching so that fixture formats sharing the same `transition_tool_cache_key` can reuse t8n results instead of calling the transition tool again.

## Opportunity

The `state_test` fixture format does not participate in t8n output caching (`transition_tool_cache_key = ""`). When a `state_test` and `blockchain_test` are filled for the same test function, the `state_test` makes its own t8n call even when the output is identical to an already-cached `blockchain_test` result.

Current execution order and t8n calls per StateTest:

| Format | t8n calls | Cache role |
| ------------------------------------------ | --------- | ---------------------- |
| `blockchain_test_from_state_test` | 1 | populates cache |
| `blockchain_test_engine_from_state_test` | 0 | cache hit |
| `state_test` | 1 | **no caching** |
| `blockchain_test_engine_x_from_state_test` | 1 | no caching (by design) |

## Why `state_test` can share cache with `blockchain_test` for Paris and Shanghai only

The t8n output for `state_test` and `blockchain_test` is identical when two conditions hold:

1. **Reward = 0** - Both formats pass reward=0 to t8n. True for all post-Merge forks (Paris onward), false for pre-Merge forks where difficulty handling differs.

2. **`pre_allocation()` == `pre_allocation_blockchain()` == `{}`** - The `state_test` format uses `pre_allocation()` while `blockchain_test` uses `pre_allocation_blockchain()`. For Paris and Shanghai, both return `{}` (no system contracts). Starting from Cancun, `pre_allocation_blockchain()` adds system contracts (beacon root via EIP-4788, then history storage, consolidation requests in Prague+) that `pre_allocation()` does not, so the t8n inputs diverge.

## Proposed implementation

### 1. Add `transition_tool_cache_forks` ClassVar to `BaseFixture`

**File:** `packages/testing/src/execution_testing/fixtures/base.py`

Add alongside `transition_tool_cache_key` (line 84):

```python
transition_tool_cache_forks: ClassVar[Sequence[type] | None] = None
```

- `None` (default): cache key is valid for all forks (existing behavior).
- Sequence of Fork classes: cache key is only valid for those forks.

### 2. Set both ClassVars on `StateFixture`

**File:** `packages/testing/src/execution_testing/fixtures/state.py`

```python
from execution_testing.forks import Fork, Paris, Shanghai

class StateFixture(BaseFixture):
...
transition_tool_cache_key: ClassVar[str] = "blockchain_test"
transition_tool_cache_forks: ClassVar[Sequence[type] | None] = (Paris, Shanghai)
```

### 3. Propagate `transition_tool_cache_forks` as a pytest marker

**File:** `packages/testing/src/execution_testing/cli/pytest_commands/plugins/shared/helpers.py`

In `labeled_format_parameter_set()`, after the existing `transition_tool_cache_key` marker block (~L49-57), add:

```python
transition_tool_cache_forks = getattr(
format_with_or_without_label, "transition_tool_cache_forks", None
)
if transition_tool_cache_forks is not None:
marks.append(
pytest.mark.transition_tool_cache_forks(transition_tool_cache_forks),
)
```

### 4. Fork-aware check in `get_t8n_cache_key()`

**File:** `packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/filler.py`

Modify `get_t8n_cache_key()` (L1201-1208) to check for a fork restriction:

```python
def get_t8n_cache_key(request: pytest.FixtureRequest) -> str | None:
"""Get the cache key to be used for the current test, if any."""
mark = request.node.get_closest_marker("transition_tool_cache_key")
if mark is None or len(mark.args) != 1:
return None
# Check fork restriction if present.
forks_mark = request.node.get_closest_marker("transition_tool_cache_forks")
if forks_mark is not None and len(forks_mark.args) == 1:
allowed_forks = forks_mark.args[0]
fork = request.node.callspec.params.get("fork")
if fork is None or fork not in allowed_forks:
return None
return f"{strip_fixture_format_from_node(request.node)}-{mark.args[0]}"
```

### 5. Add tests

**File:** `packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/tests/test_t8n_cache.py`

Add a `TestForkConditionalCaching` class testing:

- `state_test` on Paris/Shanghai - returns cache key (matches `blockchain_test`)
- `state_test` on Cancun/other forks - returns `None` (no caching)
- `blockchain_test` (no fork restriction) - returns cache key for all forks

### Sorting impact

No changes needed. `state_test` items with `transition_tool_cache_key` set will sort as cacheable (grouped with the blockchain formats). For non-Paris/Shanghai forks, `get_t8n_cache_key()` returns `None` at runtime - `remove_cache()` is called, but this is harmless because it runs AFTER the blockchain pair has already consumed the cache (`state_test` sorts last alphabetically among cacheable formats: `blockchain_test` < `blockchain_test_engine` < `state_test`).

## Expected savings

Per-test t8n calls on Paris/Shanghai for default `fill` options (no EngineX):

| Before #2084 | After #2084 | After this issue |
| ------------ | ----------- | ---------------- |
| 3 calls | 2 calls | 1 call |

## Verification plan

1. `uvx tox -e static` - type checking and linting pass
2. `uvx tox -e tests_pytest_py3,tests_benchmark_pytest_py3` - unit tests pass
3. Manual check: fill exactly one `blockchain_test` and one `state_test` for Shanghai and verify the cache summary shows a hit:

```bash
uv run fill \
"tests/berlin/eip2929_gas_cost_increases/test_call.py::test_call_insufficient_balance[fork_Shanghai-blockchain_test_from_state_test]" \
"tests/berlin/eip2929_gas_cost_increases/test_call.py::test_call_insufficient_balance[fork_Shanghai-state_test]"
```

Expect the terminal summary to report `T8n cache: 100% hit rate (1/1 tests expected), 1 t8n calls saved`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.