PolicyEngine / PolicyEngine/policyengine-api

Add explicit codecov.yml policy to unblock admin-merge workflow

Open
#3,476 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
18
Forks
33
Avg merge
23h 40m
Merged PRs (30d)
22

Description

Summary

codecov/patch enforces a 77.10% patch-coverage threshold derived from the base branch's current coverage. There is no codecov.yml in the repo, so the threshold is implicit and tracks master drift. Recent bug-fix PRs (e.g. #3454) failed this check despite adding regression tests for every change, because fixes that touch non-code paths (exception handlers, whitelist guards, return-None short-circuits) don't proportionally lift coverage.

What goes wrong

  • The check blocks merge via mergeStateStatus: BLOCKED even though mergeable: MERGEABLE — admins have to merge with --admin to proceed.
  • There is no visible threshold-policy document: contributors can't see what target they're shooting at without opening Codecov's UI.
  • The de-facto workflow (admin-merge on coverage failure) undermines the signal — if admin-merge is always fine, the check should not be required; if the check matters, the threshold should be published and tunable.

Suggested fix

Add a codecov.yml at repo root with an explicit policy:

coverage:
  status:
    project:
      default:
        target: auto
        threshold: 1%
    patch:
      default:
        target: 70%    # or whatever realistic floor matches the codebase
        informational: false

Options:

  • Target auto with a threshold of 1–2% — patch can drop coverage by at most 1–2% relative to base (forgiving for bug fixes that touch few lines).
  • Fixed 70% floor — easier to reason about; allows fixes that don't add tests to non-essential defensive paths.
  • Informational = true — keeps the signal visible without blocking merges; admin-merge workaround stops being necessary.

Document the choice in CONTRIBUTING.md so contributors know the target.

Severity

Medium — procedural friction that degrades to "admin-merge everything" in practice, which hides the coverage signal entirely.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the existing CI workflow and the repository's Codecov check behavior. Add the root codecov.yml policy and document the selected target or informational behavior in CONTRIBUTING.md; done means the policy is visible to contributors and the merge workflow reflects that choice.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, documentation, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.