paritytech / paritytech/web3-storage
Make the Rust coverage gate robust and diagnostic (filename-based exclusion + whole-crate ratio is fragile)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 13
- Forks
- 3
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 33
Description
Problem
The Rust coverage job (.github/workflows/check.yml) fails on any decrease in a
whole-crate coverage ratio, with the chain-client layer excluded by a filename
suffix (--ignore-filename-regex='(…|_subxt\.rs|command\.rs|cli\.rs|main\.rs)') and
reported as a single --summary-only number. This makes the gate fragile and
low-signal:
-
Pure refactors move the number. PR #159 (reuse one
SubxtChainClient) consolidates
challenge_responder_subxt.rs/checkpoint_coordinator_subxt.rs/
replica_sync_coordinator_subxt.rs(all excluded by_subxt\.rs) into
subxt_client.rs(~915 LOC), which the regex no longer matches. Those ~0%-covered
lines now count, so provider coverage drops 63.88% → 59.08% with zero behavior
change and the gate fails. The exclusion is keyed on a filename suffix, so any
rename/split/merge silently changes the baseline. -
The denominator is full of code that can't be unit-tested without a live chain.
Ondevtodaycheckpoint_coordinator.rs,replica_sync_coordinator.rs,
challenge_responder.rs, andreplica_sync.rsare already at 0% — they only run
against a real chain, so they're dead weight in the ratio. -
--summary-onlyhides the cause. CI prints one aggregate%; "coverage decreased"
gives no hint which file moved.
Reproduced locally
cargo llvm-cov with the CI flags reproduces dev provider coverage = 63.88%. The
per-file table shows the culprits clearly (coordinator files at 0%); the *_subxt.rs
files are excluded only by the regex. Including them (i.e. what subxt_client.rs does)
drags the total to ~59%.
Proposed improvements
Make the gate diagnostic
- Drop
--summary-only; write the full per-file coverage table to
$GITHUB_STEP_SUMMARY(and/or a sticky PR comment) so a drop names the exact file. - Upload
cargo llvm-cov --htmlas a CI artifact for line-level drill-down.
Measure the right thing
- Gate on patch/diff coverage (coverage of lines changed in the PR) instead of the
whole-crate ratio — immune to renames/refactors, and it points at untested new
lines. Options:cargo llvm-cov --lcov+diff-cover, or a service like
Codecov/Coveralls (native patch coverage + PR comments). - Alternatively/additionally compare absolute covered-line count with a small
tolerance, so a denominator change from a refactor can't fail the gate.
Remove the filename fragility
- Stop excluding by
_subxtsuffix. Exclude the chain-access layer by path/module
(e.g. achain/subdir) or#[cfg_attr(coverage, coverage(off))], so file renames
don't shift the baseline.
Immediate unblock (separate small PR)
- Add
subxt_client\.rsto the provider--ignore-filename-regexso PR #159 (a
no-behavior refactor) isn't blocked by the gate.
References
- Failing run: https://github.com/paritytech/web3-storage/actions/runs/27547487242/job/81426731122 (PR #159)
- Gate:
.github/workflows/check.yml(coveragejob — Measure/Compare coverage steps)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in .github/workflows/check.yml's coverage job, especially the Measure and Compare steps, and reproduce the current result with cargo llvm-cov using the CI flags. Review the per-file table, ignore regex, and proposed diagnostic and gating alternatives; done means the agreed coverage gate is robust to file refactors and identifies coverage changes clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, rust
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100