codecov / codecov/codecov-action

Coverage not calculated correctly if coverage report is uploaded without files being present

Open
#1,707 0 comments 0 reactions 1 assignee View on GitHub

@thomasrockhu-codecov is already working on this.

Since Feb 19, 2025.

Area: Report Processing bug codecov/codecov-action
Dominant language
Python
Stars
1.7k
Forks
260
Avg merge
1d 7h
Merged PRs (30d)
1

Description

We have recently updated our CI workflow to use sharding in Jest to speed up the tests. The flow is something like this:

  • Job 1: Checkout, install and build.
  • Then job 2 and 3 in parallel: checkout, install and run tests. Each job runs 50% of the tests. Coverage then gets uploaded as an artifact via actions/upload-artifact.
  • Job 4: download coverage with actions/download-artifact, merge into a single json and report to Codecov via codecov/codecov-action.

The problem was that job 4 didn't checkout the repo (because it didn't really need those files).
When codecov-action was used without the repo being present, even though the coverage file was uploaded, it always said All modified and coverable lines are covered by tests ✅ and all changes were reported as indirect changes.

File paths were incorrect (extra webui) and clicking on the file resulted in an error:

https://github.com/user-attachments/assets/f65bc50c-9fb4-490d-beb3-711e5d8b67ac

Example of a bad job:
https://github.com/truenas/webui/actions/runs/12073223216/job/33669795147?pr=11114

After I added repo checkout as a step, everything started to work correctly:
https://github.com/truenas/webui/actions/runs/12082374244/job/33694460349?pr=11114

You can see my previous CI configuration and fixes in the associated PR. There were no important changes in how coverage file is generated.

While this is a configuration error, I feel like there could be something to improve on Codecov's end:

  • It could have given me an error on upload if it really needed files to be present.
  • We have patch coverage target set up. It would have been better if it started failing on all jobs. Instead it was always green for a month, before we spotted that something was wrong.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.