Enforce a coverage ratchet in CI

Open
#475 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
65/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
github-actions, python

Research direction

Start by reading .github/workflows/ci.yml, the [tool.coverage.report] section in pyproject.toml, and the relevant guidance in CONTRIBUTING.md. Run make test locally and inspect the coverage output to establish the current floor, then compare the single gated matrix cell with the other CI jobs. Done means the committed floor matches measured coverage, regressions fail clearly, the policy is documented, and full-matrix runtime does not materially increase.

Written by the indexing model from the issue text.

Description

complexity:average developer-experience priority:medium testing

Summary

CI runs the test suite with --cov=contextweaver --cov-report=term-missing on every matrix cell, but nothing consumes the number: there is no fail_under, no upload/tracking service, and no diff coverage on PRs. Coverage is paid for on every run and can regress silently. This issue adds a ratcheted floor.

Why this matters

With 1,848 tests and branch coverage already configured ([tool.coverage.run] branch = true), the project is one config line away from making coverage a guarantee instead of a scrolling log artifact. A ratchet (set at the current measured level, raised deliberately) prevents the slow erosion that otherwise accompanies rapid feature addition — and several issues from this audit cycle add code in security-grade areas where untested branches are most costly.

Current evidence

  • .github/workflows/ci.yml — pytest invoked with coverage flags; no consumer of the result.
  • pyproject.toml [tool.coverage.report]show_missing, exclude_lines configured; no fail_under.
  • No codecov/coveralls integration; the floor-deps job deliberately skips --cov (fine — it proves resolvability, not behavior).

Proposed implementation

  1. Measure current line+branch coverage on main (one CI run's term-missing output or a local run).
  2. Set fail_under in [tool.coverage.report] to the measured value, rounded down to the nearest whole percent.
  3. Run the coverage-gated pytest on a single matrix cell (e.g. the 3.12 cell that already runs the extra steps) to avoid quadrupling the cost; other cells can run with --no-cov for speed if desired.
  4. Document the ratchet policy in CONTRIBUTING.md: the floor only moves up; PRs that meaningfully raise coverage may bump it.
  5. Optional follow-up (separate decision): diff coverage on PRs via diff-cover or a coverage-report job summary ($GITHUB_STEP_SUMMARY works for fork PRs, unlike sticky comments).

Acceptance criteria

  • A PR that drops total coverage below the committed floor fails CI with a clear message.
  • The floor matches measured main-branch coverage at merge time (no aspirational number).
  • CONTRIBUTING.md documents the policy.
  • Full-matrix CI wall time does not increase materially.

Test plan

  • Deliberately add an uncovered branch in a scratch branch and confirm the gate fails; revert.
  • Confirm make test locally reports the same number CI enforces.

Migration notes

Contributor workflow change: PRs adding significant untested code now fail a named gate instead of merging silently. Not expected to require other migration.

Risks and tradeoffs

  • A single global number can be gamed by adding trivial tests; it is a floor, not a target — review remains the real gate.
  • Coverage-measurement flakiness across Python versions is avoided by gating on one pinned cell.

Suggested labels

testing, developer-experience

Dominant language
Python
Stars
9
Forks
17
Avg merge
21h 36m
Merged PRs (30d)
22

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dgenio/contextweaver

All issues in dgenio/contextweaver

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.