ethereum / ethereum/execution-specs

Audit pre-Cancun GeneralStateTest coverage in execution-specs

Open
#3,404 3 comments 0 reactions 2 assignees Claimed by @spencer-tb View on GitHub
A-tests C-chore
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 8h
Merged PRs (30d)
106

Description

## Top-level objective

Our end goal is to archive `ethereum/tests` and `ethereum/legacytests` once the `execution-specs` test release provides complete replacement coverage. This issue is one audit subtask under that meta-goal: establish whether all relevant pre-Cancun GeneralStateTest scenarios are covered, identify gaps, and make the result reproducible.

Blockchain-only tests and other legacy suites will need their own audit or migration subtasks before either repository can be archived.

## Problem

[execution-spec-tests PR #1442](https://github.com/ethereum/execution-spec-tests/pull/1442) imported the active GeneralStateTests from an `ethereum/tests` snapshot and filled them from Cancun onward. Tests that had already become legacy-only were outside that input set. The Weld subsequently moved the resulting static-test collection into `execution-specs`, preserving that historical boundary.

We now have concrete evidence of a missing consensus-regression case:

- [`RevertPrecompiledTouch_d0g0v0.json`](https://github.com/ethereum/legacytests/blob/1f581b8ccdc4c63acf5f2c5c1b155c690c32a8eb/Constantinople/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch_d0g0v0.json) was generated from a GeneralStateTest, so this is not a blockchain-test-format limitation.
- The pre-Cancun source test was removed from the active `ethereum/tests` set in May 2024 and remained available through `ethereum/legacytests`.
- The similarly named test now in `execution-specs` is valid from Cancun and funds the precompile accounts, so it cannot exercise deletion of the empty RIPEMD-160 account under the historical Spurious Dragon behavior.
- EELS explicitly implements the block 2,675,119 RIPEMD-160 touched-account exception, but the unified fixture release does not currently contain the historical regression scenario that would exercise it.

This example also shows why filename matching is insufficient: a current test can share the legacy test's name and broad control flow without covering the same consensus condition.

## Goal

Produce a complete and reviewable mapping from pre-Cancun GeneralStateTest scenarios to their coverage in `execution-specs`, then create focused follow-up work for every confirmed gap. The resulting evidence should satisfy the GeneralStateTests portion of the top-level archival checklist.

## Coverage model

Use an authored scenario combined with each materially distinct fork-semantic regime as the unit of coverage. Do not count every generated data/gas/value (`d/g/v`) expansion as an independent authored test, and do not require a one-to-one file port when a modern test demonstrably exercises the same rule and assertions.

A scenario should be considered covered only when the current test reaches the same consensus behavior with equivalent material preconditions and post-state assertions on a relevant fork. Renamed or consolidated tests may qualify with evidence. Obsolete, redundant, or intentionally unsupported cases should receive an explicit rationale rather than silently disappearing from the inventory.

## Proposed work

1. Establish a canonical pre-Cancun GeneralStateTest inventory from the last complete pre-removal `ethereum/tests` source snapshot, cross-checked against the corresponding `ethereum/legacytests` state and generated blockchain fixtures.
2. Normalize each source scenario into the coverage unit above and record its relevant fork-semantic regimes, provenance, and legacy fixture references.
3. Map each unit to `execution-specs` Python tests, ported static tests, or another demonstrably equivalent current test using provenance markers first and semantic review where provenance is absent.
4. Assign every unit one disposition: directly ported, semantically covered, missing, intentionally not ported with rationale, or unresolved.
5. Emit both a machine-readable ledger and a human-readable report, including evidence for every non-trivial equivalence decision.
6. Open focused follow-up issues or PRs for confirmed gaps, starting with the historical RIPEMD-160 touched-account deletion case.
7. Make the audit reproducible so it can be rerun as current tests change and used as an archival gate rather than a one-time spreadsheet.

## Deliverables

- A versioned inventory of pre-Cancun GeneralStateTest scenario/fork-semantic units with source provenance.
- A machine-readable mapping from every inventory unit to its `execution-specs` disposition and supporting evidence.
- A summarized coverage report listing confirmed gaps, justified exclusions, and unresolved cases.
- Focused follow-up issues for confirmed missing coverage.
- A documented, rerunnable command or workflow that refreshes the report.

## Acceptance criteria

- Every pre-Cancun GeneralStateTest unit in the canonical inventory has exactly one disposition and a supporting source or coverage reference.
- Semantic matches are reviewed against material preconditions, executed behavior, fork validity, and post-state assertions rather than accepted by filename alone.
- The known `RevertPrecompiledTouch` empty-RIPEMD-account scenario is reported as missing unless equivalent historical-fork coverage is added before the audit lands.
- Every confirmed gap has a focused follow-up issue or linked implementation PR; every intentional exclusion has a written rationale.
- Any unresolved unit remains an explicit blocker for the GeneralStateTests archival gate.
- The top-level repository-archival tracker is updated with the report and the resulting GeneralStateTests gate status.

## Out of scope

- Auditing blockchain-only scenarios or non-GeneralStateTest suites such as transaction, difficulty, RLP, and VM tests; these require sibling subtasks.
- Porting every confirmed gap within this audit issue.
- Archiving either legacy repository before all suite-level coverage gates are complete.
- Requiring byte-for-byte fixture identity when semantic equivalence can be demonstrated.

## Acknowledgements

H/t @taratorio and @chfast for flagging the broader risk of missing pre-Cancun coverage.

## Related evidence

- [Historical legacy fixture](https://github.com/ethereum/legacytests/blob/1f581b8ccdc4c63acf5f2c5c1b155c690c32a8eb/Constantinople/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch_d0g0v0.json)
- [GeneralStateTests integration in execution-spec-tests](https://github.com/ethereum/execution-spec-tests/pull/1442)
- [Yellow Paper Appendix K.1](https://ethereum.github.io/yellowpaper/paper.pdf)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.