ethereum / ethereum/execution-specs
Audit pre-Cancun GeneralStateTest coverage in execution-specs
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 106
Description
## Top-level objective
Our end goal is to archive `ethereum/tests` and `ethereum/legacytests` once the `execution-specs` test release provides complete replacement coverage. This issue is one audit subtask under that meta-goal: establish whether all relevant pre-Cancun GeneralStateTest scenarios are covered, identify gaps, and make the result reproducible.
Blockchain-only tests and other legacy suites will need their own audit or migration subtasks before either repository can be archived.
## Problem
[execution-spec-tests PR #1442](https://github.com/ethereum/execution-spec-tests/pull/1442) imported the active GeneralStateTests from an `ethereum/tests` snapshot and filled them from Cancun onward. Tests that had already become legacy-only were outside that input set. The Weld subsequently moved the resulting static-test collection into `execution-specs`, preserving that historical boundary.
We now have concrete evidence of a missing consensus-regression case:
- [`RevertPrecompiledTouch_d0g0v0.json`](https://github.com/ethereum/legacytests/blob/1f581b8ccdc4c63acf5f2c5c1b155c690c32a8eb/Constantinople/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch_d0g0v0.json) was generated from a GeneralStateTest, so this is not a blockchain-test-format limitation.
- The pre-Cancun source test was removed from the active `ethereum/tests` set in May 2024 and remained available through `ethereum/legacytests`.
- The similarly named test now in `execution-specs` is valid from Cancun and funds the precompile accounts, so it cannot exercise deletion of the empty RIPEMD-160 account under the historical Spurious Dragon behavior.
- EELS explicitly implements the block 2,675,119 RIPEMD-160 touched-account exception, but the unified fixture release does not currently contain the historical regression scenario that would exercise it.
This example also shows why filename matching is insufficient: a current test can share the legacy test's name and broad control flow without covering the same consensus condition.
## Goal
Produce a complete and reviewable mapping from pre-Cancun GeneralStateTest scenarios to their coverage in `execution-specs`, then create focused follow-up work for every confirmed gap. The resulting evidence should satisfy the GeneralStateTests portion of the top-level archival checklist.
## Coverage model
Use an authored scenario combined with each materially distinct fork-semantic regime as the unit of coverage. Do not count every generated data/gas/value (`d/g/v`) expansion as an independent authored test, and do not require a one-to-one file port when a modern test demonstrably exercises the same rule and assertions.
A scenario should be considered covered only when the current test reaches the same consensus behavior with equivalent material preconditions and post-state assertions on a relevant fork. Renamed or consolidated tests may qualify with evidence. Obsolete, redundant, or intentionally unsupported cases should receive an explicit rationale rather than silently disappearing from the inventory.
## Proposed work
1. Establish a canonical pre-Cancun GeneralStateTest inventory from the last complete pre-removal `ethereum/tests` source snapshot, cross-checked against the corresponding `ethereum/legacytests` state and generated blockchain fixtures.
2. Normalize each source scenario into the coverage unit above and record its relevant fork-semantic regimes, provenance, and legacy fixture references.
3. Map each unit to `execution-specs` Python tests, ported static tests, or another demonstrably equivalent current test using provenance markers first and semantic review where provenance is absent.
4. Assign every unit one disposition: directly ported, semantically covered, missing, intentionally not ported with rationale, or unresolved.
5. Emit both a machine-readable ledger and a human-readable report, including evidence for every non-trivial equivalence decision.
6. Open focused follow-up issues or PRs for confirmed gaps, starting with the historical RIPEMD-160 touched-account deletion case.
7. Make the audit reproducible so it can be rerun as current tests change and used as an archival gate rather than a one-time spreadsheet.
## Deliverables
- A versioned inventory of pre-Cancun GeneralStateTest scenario/fork-semantic units with source provenance.
- A machine-readable mapping from every inventory unit to its `execution-specs` disposition and supporting evidence.
- A summarized coverage report listing confirmed gaps, justified exclusions, and unresolved cases.
- Focused follow-up issues for confirmed missing coverage.
- A documented, rerunnable command or workflow that refreshes the report.
## Acceptance criteria
- Every pre-Cancun GeneralStateTest unit in the canonical inventory has exactly one disposition and a supporting source or coverage reference.
- Semantic matches are reviewed against material preconditions, executed behavior, fork validity, and post-state assertions rather than accepted by filename alone.
- The known `RevertPrecompiledTouch` empty-RIPEMD-account scenario is reported as missing unless equivalent historical-fork coverage is added before the audit lands.
- Every confirmed gap has a focused follow-up issue or linked implementation PR; every intentional exclusion has a written rationale.
- Any unresolved unit remains an explicit blocker for the GeneralStateTests archival gate.
- The top-level repository-archival tracker is updated with the report and the resulting GeneralStateTests gate status.
## Out of scope
- Auditing blockchain-only scenarios or non-GeneralStateTest suites such as transaction, difficulty, RLP, and VM tests; these require sibling subtasks.
- Porting every confirmed gap within this audit issue.
- Archiving either legacy repository before all suite-level coverage gates are complete.
- Requiring byte-for-byte fixture identity when semantic equivalence can be demonstrated.
## Acknowledgements
H/t @taratorio and @chfast for flagging the broader risk of missing pre-Cancun coverage.
## Related evidence
- [Historical legacy fixture](https://github.com/ethereum/legacytests/blob/1f581b8ccdc4c63acf5f2c5c1b155c690c32a8eb/Constantinople/BlockchainTests/GeneralStateTests/stRevertTest/RevertPrecompiledTouch_d0g0v0.json)
- [GeneralStateTests integration in execution-spec-tests](https://github.com/ethereum/execution-spec-tests/pull/1442)
- [Yellow Paper Appendix K.1](https://ethereum.github.io/yellowpaper/paper.pdf)
Contributor guide
Assessment
This issue has not been assessed yet.