ethereum-optimism / ethereum-optimism/optimism

ci(rust): re-enable kona coverage

Open
#20,670 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

## Context

`kona-coverage` was disabled in #20667 and the dead code removed in the follow-up cleanup PR. Before disabling, the job was alternating between fake-greens (uploading a near-empty lcov to Codecov) and 40-minute `no_output_timeout` failures — see the analysis in #20667. We want coverage back, but only once the recipe is rebuilt to be reliable.

This issue tracks what needs to land to re-enable it.

## What was broken (root causes)

Recapping the diagnosis from #20667 so re-enabling doesn't repeat history:

1. **OOM on the CI executor.** With `-C instrument-coverage --cfg=coverage` and `--all-features` enabled across the entire `rust/` workspace, rustc is OOM-killed (SIGKILL) while compiling `reth-optimism-node` on the `xlarge` Docker class (8 vCPU / 16 GB).
2. **`set -e` was missing.** `rust/kona/justfile`'s `llvm-cov-tests` recipe used `#!/usr/bin/env bash` with no `set -euo pipefail`. After the workspace build failed, bash kept going, ran a tiny `kona-registry` test on its own, generated an `lcov.info` covering only that test, and exited 0. Codecov then ingested garbage.
3. **Stale `--exclude kona-p2p`.** `kona-p2p` no longer exists; the package was split into `kona-gossip` / `kona-disc` / `kona-peers` / `kona-node-service`. Produced a warning every run.
4. **Cache mis-pointing.** `rust-prepare-and-restore-cache` saves/restores `rust/target/debug`, but `cargo llvm-cov` writes to `rust/target/llvm-cov-target`. Every run was effectively cold-build, contributing to the timeout failures and to the OOM (the kernel gets less patient under memory pressure with no warm dep graph).
5. **Wrong Codecov flag.** The upload step used `flags: unit`, but `codecov.yml` only knows about `kona-unit` (see `codecov.yml:33,45,58`). So even when an upload succeeded, the `kona-unit` Codecov status check stayed grey.
6. **Orphan instrumentation in `kona-host-client-offline`.** The job was sourcing `cargo llvm-cov show-env` and writing `client_host_cov.lcov`, but no upload step consumed the file. Removed in the cleanup PR.

## Re-enable checklist

- [ ] Pick a strategy for fitting the build into memory. Options, roughly in order of effort:
- Bump the executor to `2xlarge` (16 vCPU / 32 GB). Cheapest fix, may still be tight when reth grows.
- Cap parallelism: `CARGO_BUILD_JOBS=4`, `-C codegen-units=1` or similar so peak memory per rustc invocation drops. Trades wall-clock for headroom.
- Narrow scope: drop `--workspace --all-features` and instrument only kona (`--manifest-path rust/kona/...`) plus optional follow-up jobs for op-reth, op-alloy, etc. Aligns with how the `kona-unit` flag is already scoped in `codecov.yml`.
- Split into multiple parallel coverage jobs (kona / op-reth / op-alloy / op-revm) merged at upload time using Codecov's flag merging — closer to what the flag layout in `codecov.yml` seems to anticipate.
- [ ] `set -euo pipefail` (or rewrite as a sequence of just commands so each step's failure surfaces). Verify by deliberately failing one cargo invocation and confirming the job goes red.
- [ ] Audit the workspace-level `--exclude` list against `rust/Cargo.toml`'s actual members. Drop `kona-p2p`, add anything else that legitimately needs excluding (e.g. crates that don't build with coverage instrumentation under `--all-features`).
- [ ] Fix the cache. Either save `rust/target/llvm-cov-target` (and align the `version` with other rust jobs so the registry cache is shared), or use `cargo llvm-cov`'s `--target-dir` to point it at `target/debug` so the existing cache works.
- [ ] Rename `flags: unit` → `flags: kona-unit` (matches `codecov.yml`). Confirm the `kona-unit` patch status flips from grey to green/red on a test PR.
- [ ] Decide whether to instrument and upload coverage from `kona-host-client-offline` too. If yes, restore the `cargo llvm-cov show-env` setup and add a `codecov/upload` step with a sensible flag (e.g. `kona-host-client`). If no, leave it as-is.
- [ ] Wire the new job back into the `rust-ci` workflow and into `required-rust-ci`'s fan-in (it was at `kona-coverage: terminal` between `kona-build-fpvm-cannon-client` and `kona-host-client-offline-cannon`).

## Validation

A re-enable PR should demonstrate, on its own CI run:

- The coverage job completes in well under 40 minutes with no OOM.
- The uploaded `lcov.info` covers the bulk of kona crates (not just one test). Eyeball line counts before/after.
- The `kona-unit` Codecov status check posts a non-grey result.
- `required-rust-ci` still fans in on the new job.

## Related

- #20667 — disable kona-coverage in CI
- (cleanup PR) — remove dead kona-coverage job code and the orphan `cargo llvm-cov` setup in `kona-host-client-offline`

🤖 *Generated by Claude Code*

Contributor guide

Open the contributing guide

Research direction

Start with rust/kona/justfile, rust/Cargo.toml, codecov.yml, and the rust-ci and required-rust-ci workflow definitions. Review the prior disablement in #20667, choose and validate a memory-safe coverage strategy, then verify failure propagation, cache behavior, Codecov flags, coverage breadth, runtime, and required-rust-ci fan-in on a CI run.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
build-system, ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.