paritytech / paritytech/web3-storage

Make the Rust coverage gate robust and diagnostic (filename-based exclusion + whole-crate ratio is fragile)

Open
#176 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug enhancement
Dominant language
Rust
Stars
13
Forks
3
Avg merge
2d 2h
Merged PRs (30d)
33

Description

Problem

The Rust coverage job (.github/workflows/check.yml) fails on any decrease in a
whole-crate coverage ratio, with the chain-client layer excluded by a filename
suffix
(--ignore-filename-regex='(…|_subxt\.rs|command\.rs|cli\.rs|main\.rs)') and
reported as a single --summary-only number. This makes the gate fragile and
low-signal:

  1. Pure refactors move the number. PR #159 (reuse one SubxtChainClient) consolidates
    challenge_responder_subxt.rs / checkpoint_coordinator_subxt.rs /
    replica_sync_coordinator_subxt.rs (all excluded by _subxt\.rs) into
    subxt_client.rs (~915 LOC), which the regex no longer matches. Those ~0%-covered
    lines now count, so provider coverage drops 63.88% → 59.08% with zero behavior
    change and the gate fails. The exclusion is keyed on a filename suffix, so any
    rename/split/merge silently changes the baseline.

  2. The denominator is full of code that can't be unit-tested without a live chain.
    On dev today checkpoint_coordinator.rs, replica_sync_coordinator.rs,
    challenge_responder.rs, and replica_sync.rs are already at 0% — they only run
    against a real chain, so they're dead weight in the ratio.

  3. --summary-only hides the cause. CI prints one aggregate %; "coverage decreased"
    gives no hint which file moved.

Reproduced locally

cargo llvm-cov with the CI flags reproduces dev provider coverage = 63.88%. The
per-file table shows the culprits clearly (coordinator files at 0%); the *_subxt.rs
files are excluded only by the regex. Including them (i.e. what subxt_client.rs does)
drags the total to ~59%.

Proposed improvements

Make the gate diagnostic

  • Drop --summary-only; write the full per-file coverage table to
    $GITHUB_STEP_SUMMARY (and/or a sticky PR comment) so a drop names the exact file.
  • Upload cargo llvm-cov --html as a CI artifact for line-level drill-down.

Measure the right thing

  • Gate on patch/diff coverage (coverage of lines changed in the PR) instead of the
    whole-crate ratio — immune to renames/refactors, and it points at untested new
    lines. Options: cargo llvm-cov --lcov + diff-cover, or a service like
    Codecov/Coveralls (native patch coverage + PR comments).
  • Alternatively/additionally compare absolute covered-line count with a small
    tolerance, so a denominator change from a refactor can't fail the gate.

Remove the filename fragility

  • Stop excluding by _subxt suffix. Exclude the chain-access layer by path/module
    (e.g. a chain/ subdir) or #[cfg_attr(coverage, coverage(off))], so file renames
    don't shift the baseline.

Immediate unblock (separate small PR)

  • Add subxt_client\.rs to the provider --ignore-filename-regex so PR #159 (a
    no-behavior refactor) isn't blocked by the gate.

References

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in .github/workflows/check.yml's coverage job, especially the Measure and Compare steps, and reproduce the current result with cargo llvm-cov using the CI flags. Review the per-file table, ignore regex, and proposed diagnostic and gating alternatives; done means the agreed coverage gate is robust to file refactors and identifies coverage changes clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.