ethereum-optimism / ethereum-optimism/optimism
ci(rust): re-enable kona coverage
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
## Context
`kona-coverage` was disabled in #20667 and the dead code removed in the follow-up cleanup PR. Before disabling, the job was alternating between fake-greens (uploading a near-empty lcov to Codecov) and 40-minute `no_output_timeout` failures — see the analysis in #20667. We want coverage back, but only once the recipe is rebuilt to be reliable.
This issue tracks what needs to land to re-enable it.
## What was broken (root causes)
Recapping the diagnosis from #20667 so re-enabling doesn't repeat history:
1. **OOM on the CI executor.** With `-C instrument-coverage --cfg=coverage` and `--all-features` enabled across the entire `rust/` workspace, rustc is OOM-killed (SIGKILL) while compiling `reth-optimism-node` on the `xlarge` Docker class (8 vCPU / 16 GB).
2. **`set -e` was missing.** `rust/kona/justfile`'s `llvm-cov-tests` recipe used `#!/usr/bin/env bash` with no `set -euo pipefail`. After the workspace build failed, bash kept going, ran a tiny `kona-registry` test on its own, generated an `lcov.info` covering only that test, and exited 0. Codecov then ingested garbage.
3. **Stale `--exclude kona-p2p`.** `kona-p2p` no longer exists; the package was split into `kona-gossip` / `kona-disc` / `kona-peers` / `kona-node-service`. Produced a warning every run.
4. **Cache mis-pointing.** `rust-prepare-and-restore-cache` saves/restores `rust/target/debug`, but `cargo llvm-cov` writes to `rust/target/llvm-cov-target`. Every run was effectively cold-build, contributing to the timeout failures and to the OOM (the kernel gets less patient under memory pressure with no warm dep graph).
5. **Wrong Codecov flag.** The upload step used `flags: unit`, but `codecov.yml` only knows about `kona-unit` (see `codecov.yml:33,45,58`). So even when an upload succeeded, the `kona-unit` Codecov status check stayed grey.
6. **Orphan instrumentation in `kona-host-client-offline`.** The job was sourcing `cargo llvm-cov show-env` and writing `client_host_cov.lcov`, but no upload step consumed the file. Removed in the cleanup PR.
## Re-enable checklist
- [ ] Pick a strategy for fitting the build into memory. Options, roughly in order of effort:
- Bump the executor to `2xlarge` (16 vCPU / 32 GB). Cheapest fix, may still be tight when reth grows.
- Cap parallelism: `CARGO_BUILD_JOBS=4`, `-C codegen-units=1` or similar so peak memory per rustc invocation drops. Trades wall-clock for headroom.
- Narrow scope: drop `--workspace --all-features` and instrument only kona (`--manifest-path rust/kona/...`) plus optional follow-up jobs for op-reth, op-alloy, etc. Aligns with how the `kona-unit` flag is already scoped in `codecov.yml`.
- Split into multiple parallel coverage jobs (kona / op-reth / op-alloy / op-revm) merged at upload time using Codecov's flag merging — closer to what the flag layout in `codecov.yml` seems to anticipate.
- [ ] `set -euo pipefail` (or rewrite as a sequence of just commands so each step's failure surfaces). Verify by deliberately failing one cargo invocation and confirming the job goes red.
- [ ] Audit the workspace-level `--exclude` list against `rust/Cargo.toml`'s actual members. Drop `kona-p2p`, add anything else that legitimately needs excluding (e.g. crates that don't build with coverage instrumentation under `--all-features`).
- [ ] Fix the cache. Either save `rust/target/llvm-cov-target` (and align the `version` with other rust jobs so the registry cache is shared), or use `cargo llvm-cov`'s `--target-dir` to point it at `target/debug` so the existing cache works.
- [ ] Rename `flags: unit` → `flags: kona-unit` (matches `codecov.yml`). Confirm the `kona-unit` patch status flips from grey to green/red on a test PR.
- [ ] Decide whether to instrument and upload coverage from `kona-host-client-offline` too. If yes, restore the `cargo llvm-cov show-env` setup and add a `codecov/upload` step with a sensible flag (e.g. `kona-host-client`). If no, leave it as-is.
- [ ] Wire the new job back into the `rust-ci` workflow and into `required-rust-ci`'s fan-in (it was at `kona-coverage: terminal` between `kona-build-fpvm-cannon-client` and `kona-host-client-offline-cannon`).
## Validation
A re-enable PR should demonstrate, on its own CI run:
- The coverage job completes in well under 40 minutes with no OOM.
- The uploaded `lcov.info` covers the bulk of kona crates (not just one test). Eyeball line counts before/after.
- The `kona-unit` Codecov status check posts a non-grey result.
- `required-rust-ci` still fans in on the new job.
## Related
- #20667 — disable kona-coverage in CI
- (cleanup PR) — remove dead kona-coverage job code and the orphan `cargo llvm-cov` setup in `kona-host-client-offline`
🤖 *Generated by Claude Code*
Contributor guide
Research direction
Start with rust/kona/justfile, rust/Cargo.toml, codecov.yml, and the rust-ci and required-rust-ci workflow definitions. Review the prior disablement in #20667, choose and validate a memory-safe coverage strategy, then verify failure propagation, cache behavior, Codecov flags, coverage breadth, runtime, and required-rust-ci fan-in on a CI run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, rust
- Domain
- build-system, ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100