ethereum / ethereum/execution-specs

First Class Ported Static Tests Tracker

Open
#3,272 1 comment 0 reactions 1 assignee Claimed by @spencer-tb View on GitHub
C-tracker
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

### Context

The 2,095 machine-ported tests in `tests/ported_static/` started as flat `stXxx/` directories with a blanket `valid_from("Cancun")` and, on Amsterdam, a 153-entry skip list. The end goal of this tracker is to make them first class native tests, fork-organized, minimally floored, fork-robust, and ultimately merged into the `tests//` structure properly.

**Aim:** the overarching aim is to make it as though it looks like they never existed, move all the ported static tests into `tests/` and fill them all as low as there floor allows for. Ideally from `Frontier` for most of them.

### Phases

Note all PRs rely on each parents merging, i.e one after the other, they are built upon each other.

#### Removing The Ported Static Skip List
- [x] Part 1 — enhance & un-skip Amsterdam ported static tests: #3215
- [ ] Part 2 — un-skip the remaining 130, empty and remove the skip list + conftest machinery: #3264

#### Restructuring
- [ ] Fork-folder reorg, pure moves into `tests/ported_static///` (subject fork, decoupled from `valid_from`; layout enforced by conftest; fill-verified inert): spencer-tb/execution-specs#33
- [ ] Empirical `valid_from` floors, `protected=fork.supports_protected_txs()` for 1,716 tests + 1,690 markers lowered (1,128 now floor at Frontier; 119,610 full-range cases green; minimality audited — 392 floors provably equal their newest-used-opcode bound, 0 defects): spencer-tb/execution-specs#34
- [ ] Gas-limit hygiene — 226 non-load-bearing hardcoded tx gas limits removed (individually gated, 91 load-bearing candidates rolled back): spencer-tb/execution-specs#35

These 3 PRs will be retargeted to `forks/amsterdam` sequentially as their bases merge.

**Snapshot floor parity** (required before archiving `ethereum/tests` + `ethereum/legacytests`, see #2556)

An audit of every ported test's floor against the per-fork `post` sections of the frozen snapshots (`ethereum/tests` HEAD tarball + `legacytests/Constantinople` + `legacytests/Cancun`) shows 1,403 tests at parity or better, and **670 tests where the snapshots covered lower forks than we currently do** — archiving today would lose that coverage. Largest classes: Cancun-floored fragile tests whose originals reached Byzantium/Frontier/Istanbul, and TangerineWhistle/SpuriousDragon-floored tests whose originals expressed *per-fork* expectations (reaching parity there needs fork-conditional posts, not just lower markers).

- [ ] Close the 670 floor-parity regressions (per-file targets in the audit CSV)
- [ ] Re-run the parity audit clean before any archive decision

**Remaining enhancement queue** (floors are code-bound, not subject-bound; the `enhance-ported-test` recipes apply)
- [ ] 48 gas-snapshot tests → CodeGasMeasure rewrites
- [ ] 63 + 91 fragile tests (hand triage + load-bearing-limit files)
- [ ] Documented honest floors, no change planned: 98 TangerineWhistle (forward-all gas shape), 47 SpuriousDragon (EIP-161 fundamentals), 18 Berlin (EIP-2929 metadata), 19 London (EIP-3529 refund semantics)

**Endgame**
- [ ] Rename suites/files to native conventions (`stEIP3855_push0` → `eip3855_push0`, meaningful names for the random/fuzzed blobs where feasible)
- [ ] Merge into the native `tests//` structure and dissolve `tests/ported_static/` (reference-spec wiring, `ported_from` markers retained for `ethereum/tests` traceability)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.