ethereum / ethereum/execution-specs
feat(ci): nightly fills of active dev-fork branches
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
## Context
#3100 pointed the scheduled nightly at the mainnet `tests` feature (`--until=BPO4`) so the rotating `fixtures_tests` artifact is a release-ready `tests@` candidate. That deliberately dropped the dev-fork coverage the interim `nightly` feature had (`--until=Amsterdam`): no scheduled run fills the active dev fork any more (all tests, slow included, all fixture formats). The dev fork is where fill regressions are most likely to land, and per-PR CI only runs a trimmed fill without `slow` tests.
This is acceptable right now: Amsterdam fork development is late-stage and the `tests` feature's `--until` will bump to `Amsterdam` soon, restoring coverage. The gap re-opens whenever the next dev-fork branch (e.g. `forks/hegota`) starts: this issue tracks closing it structurally.
## Proposal
Keep the `tests` nightly exactly as is and add scheduled dev-fork fills parametrized by branch, e.g. a `nightly-dev-forks` entry in `feature.yaml` that lists the active dev-fork branches:
```yaml
# Filled nightly per active dev-fork branch; never tagged or released.
nightly-dev-forks:
evm-type: eels
branches:
forks/amsterdam: --until=Amsterdam --generate-all-formats
# forks/hegota: --until=Hegota --generate-all-formats
```
The per-branch `--until` could alternatively be derived from the branch suffix, but an explicit map keeps `generate_build_matrix.py` trivial and makes fork rotation a one-line config edit.
## Design considerations
- **Trigger shape**: the lightest option is a second cron line, dispatched on `github.event.schedule` (e.g. 02:00 for `tests`, 04:00 for the dev leg), keeping each scheduled run single-feature and staggering runner load. Fanning both features out of one scheduled run is cleaner conceptually but needs per-entry checkout refs in `build` and a per-feature `combine` (its `feature_name`/`combine_labels` outputs are scalars today).
- **Commit gate**: `check_new_commits.py` anchors on the checked-out branch's `GITHUB_SHA` against the last scheduled run that uploaded artifacts, workflow-global. With two legs, each would advance the baseline the other sees. Either discriminate legs by artifact name in `last_real_nightly()` (this interacts with the artifact-liveness anchoring from #3100), or let the dev leg skip the gate and fill unconditionally: the active fork changes most days anyway.
- **Combine is optional for the dev leg**: if the goal is "ensure all tests fill", the per-range job results are the signal and the leg could stop after `build`. A combined Amsterdam artifact has consumption value too (validating clients against incoming dev-fork tests), so leaning towards keeping it.
- **Fork-order lookup**: scheduled runs execute the default branch's scripts, so an `--until` fork missing from `FORK_ORDER` in the default branch's `generate_build_matrix.py` would `KeyError` in `applicable_ranges`. The follow-up must handle unknown forks explicitly (fail with a clear message, or fill unsplit).
- **Promotion safety**: `promote_nightly.py` selects candidates by the `fixtures_tests` artifact name, so dev-fork artifacts (e.g. `fixtures_nightly-dev-forks`) can never be promoted to a mainnet release. Keep that name-based discrimination when naming the dev-leg artifacts.
- **Runner load**: a full dev-fork fill roughly doubles nightly gigachungus usage; stagger the cron and reuse the existing skip/refresh logic where possible.
## Related
- #3100: nightly mainnet fill via the release workflow (dropped dev-fork coverage, flagged as a deliberate follow-up).
Contributor guide
Research direction
Start with feature.yaml and generate_build_matrix.py, then trace the scheduled workflow's build and combine paths. Check check_new_commits.py and promote_nightly.py before choosing how each leg identifies its baseline and artifacts. Done means active dev-fork branches fill nightly without affecting mainnet promotion, with fork-order and runner-load behavior handled clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, devops, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100