ethereum / ethereum/execution-specs

feat(ci): nightly fills of active dev-fork branches

Open
#3,178 2 comments 0 reactions 0 assignees View on GitHub
A-ci C-feat stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

## Context

#3100 pointed the scheduled nightly at the mainnet `tests` feature (`--until=BPO4`) so the rotating `fixtures_tests` artifact is a release-ready `tests@` candidate. That deliberately dropped the dev-fork coverage the interim `nightly` feature had (`--until=Amsterdam`): no scheduled run fills the active dev fork any more (all tests, slow included, all fixture formats). The dev fork is where fill regressions are most likely to land, and per-PR CI only runs a trimmed fill without `slow` tests.

This is acceptable right now: Amsterdam fork development is late-stage and the `tests` feature's `--until` will bump to `Amsterdam` soon, restoring coverage. The gap re-opens whenever the next dev-fork branch (e.g. `forks/hegota`) starts: this issue tracks closing it structurally.

## Proposal

Keep the `tests` nightly exactly as is and add scheduled dev-fork fills parametrized by branch, e.g. a `nightly-dev-forks` entry in `feature.yaml` that lists the active dev-fork branches:

```yaml
# Filled nightly per active dev-fork branch; never tagged or released.
nightly-dev-forks:
evm-type: eels
branches:
forks/amsterdam: --until=Amsterdam --generate-all-formats
# forks/hegota: --until=Hegota --generate-all-formats
```

The per-branch `--until` could alternatively be derived from the branch suffix, but an explicit map keeps `generate_build_matrix.py` trivial and makes fork rotation a one-line config edit.

## Design considerations

- **Trigger shape**: the lightest option is a second cron line, dispatched on `github.event.schedule` (e.g. 02:00 for `tests`, 04:00 for the dev leg), keeping each scheduled run single-feature and staggering runner load. Fanning both features out of one scheduled run is cleaner conceptually but needs per-entry checkout refs in `build` and a per-feature `combine` (its `feature_name`/`combine_labels` outputs are scalars today).
- **Commit gate**: `check_new_commits.py` anchors on the checked-out branch's `GITHUB_SHA` against the last scheduled run that uploaded artifacts, workflow-global. With two legs, each would advance the baseline the other sees. Either discriminate legs by artifact name in `last_real_nightly()` (this interacts with the artifact-liveness anchoring from #3100), or let the dev leg skip the gate and fill unconditionally: the active fork changes most days anyway.
- **Combine is optional for the dev leg**: if the goal is "ensure all tests fill", the per-range job results are the signal and the leg could stop after `build`. A combined Amsterdam artifact has consumption value too (validating clients against incoming dev-fork tests), so leaning towards keeping it.
- **Fork-order lookup**: scheduled runs execute the default branch's scripts, so an `--until` fork missing from `FORK_ORDER` in the default branch's `generate_build_matrix.py` would `KeyError` in `applicable_ranges`. The follow-up must handle unknown forks explicitly (fail with a clear message, or fill unsplit).
- **Promotion safety**: `promote_nightly.py` selects candidates by the `fixtures_tests` artifact name, so dev-fork artifacts (e.g. `fixtures_nightly-dev-forks`) can never be promoted to a mainnet release. Keep that name-based discrimination when naming the dev-leg artifacts.
- **Runner load**: a full dev-fork fill roughly doubles nightly gigachungus usage; stagger the cron and reuse the existing skip/refresh logic where possible.

## Related

- #3100: nightly mainnet fill via the release workflow (dropped dev-fork coverage, flagged as a deliberate follow-up).

Contributor guide

Open the contributing guide

Research direction

Start with feature.yaml and generate_build_matrix.py, then trace the scheduled workflow's build and combine paths. Check check_new_commits.py and promote_nightly.py before choosing how each leg identifies its baseline and artifacts. Done means active dev-fork branches fill nightly without affecting mainnet promotion, with fork-order and runner-load behavior handled clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, devops, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.