codecov / codecov/codecov-action
Coverage not calculated correctly if coverage report is uploaded without files being present
@thomasrockhu-codecov is already working on this.
Since Feb 19, 2025.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 260
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 1
Description
We have recently updated our CI workflow to use sharding in Jest to speed up the tests. The flow is something like this:
- Job 1: Checkout, install and build.
- Then job 2 and 3 in parallel: checkout, install and run tests. Each job runs 50% of the tests. Coverage then gets uploaded as an artifact via
actions/upload-artifact. - Job 4: download coverage with
actions/download-artifact, merge into a single json and report to Codecov viacodecov/codecov-action.
The problem was that job 4 didn't checkout the repo (because it didn't really need those files).
When codecov-action was used without the repo being present, even though the coverage file was uploaded, it always said All modified and coverable lines are covered by tests ✅ and all changes were reported as indirect changes.
File paths were incorrect (extra webui) and clicking on the file resulted in an error:
https://github.com/user-attachments/assets/f65bc50c-9fb4-490d-beb3-711e5d8b67ac
Example of a bad job:
https://github.com/truenas/webui/actions/runs/12073223216/job/33669795147?pr=11114
After I added repo checkout as a step, everything started to work correctly:
https://github.com/truenas/webui/actions/runs/12082374244/job/33694460349?pr=11114
You can see my previous CI configuration and fixes in the associated PR. There were no important changes in how coverage file is generated.
While this is a configuration error, I feel like there could be something to improve on Codecov's end:
- It could have given me an error on upload if it really needed files to be present.
- We have patch coverage target set up. It would have been better if it started failing on all jobs. Instead it was always green for a month, before we spotted that something was wrong.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.