ethereum / ethereum/execution-specs
valid_at-pinned tests are silently excluded when filling against a development fork
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
Filling the whole suite against a development fork silently drops tests, with no signal that coverage is short.
### The mechanism
`valid_from(X)` builds its fork set by comparison — `{f for f in ALL_FORKS if f >= X}` — so `BinaryTree`, which subclasses `Amsterdam`, is included. `valid_at(X)` instead resolves its arguments to *literally those forks*. `BinaryTree` is never literally named, the intersection with the selected fork set is empty, and an empty intersection means the test is simply not parametrized: no skip, no warning, nothing at default verbosity.
### Scale
13 `valid_at` usages resolve to Amsterdam and therefore *should* run against `BinaryTree` (Amsterdam semantics plus a commitment swap), but don't:
| Marker | Usages |
|---|---|
| `valid_at("EIP7981")` | 4 |
| `valid_at("EIP7976")` | 2 |
| `valid_at("Amsterdam")` | 2 |
| `valid_at("EIP8037")`, `("EIP7997")`, `("EIP7954")`, `("EIP7843")`, `("EIP7708")` | 1 each |
The remaining `valid_at` usages (`"Prague"`, `"Osaka"`, `"Homestead"`) are correctly excluded.
### The sharper problem
13 missing tests is tolerable; a port that drops tests with no signal is not, because nobody running `just binary-trie-fork tests` can tell coverage is short.
### Possible directions
1. **Report exclusions** — have a fork-scoped fill print what was deselected by exact-set markers. Additive, no blast radius, turns an invisible gap into a visible one. Cheapest real improvement.
2. **Name the fork explicitly** in those 13 markers. Works, but edits upstream tests for an experimental fork and doesn't scale — the next development fork hits the same wall.
3. **Change `ValidAt` semantics** to expand along the subclass axis. Fixes it everywhere at once, but `valid_at` currently means "exactly this fork" and is used suite-wide, so this needs an owner's decision on the marker contract.
The broader goal is that the entire suite should eventually be runnable against PBT, so whatever is chosen should generalise to any future development fork rather than special-casing `BinaryTree`.
Surfaced by #3246, which documents this as a known gap.
Contributor guide
Research direction
Start by running `just binary-trie-fork tests` and inspect the `valid_from`, `valid_at`, and `ValidAt` marker behavior described in the issue. Review #3246 for the known-gap context, then determine which fork-selection contract is approved. Done means the suite no longer hides excluded tests and the behavior generalizes to future development forks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100