check-coverage passes when there is no coverage data to check
- Dominant language
- JavaScript
- Stars
- 2.1k
- Forks
- 99
- PR merge metrics
- No merged PRs in 30d
Description
* **Version**: 12.0.0, read from `node_modules/c8/package.json` (`c8 --version` prints `1.0.0`, which is #607)
* **Platform**: Windows 11 (10.0.26200), 64-bit, Node v24.15.0
`check-coverage` exits 0 when there is no coverage data to check. The report for the same run says `Unknown% ( 0/0 )`, so the fact that nothing was measured is known at report time and does not reach the threshold check.
### Reproduction
Fresh directory, `npm init -y`, `npm i c8@12.0.0`.
1. A run that produces no coverage data:
```
$ npx c8 --check-coverage --lines 100 node -e "1"
----------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files | 0 | 0 | 0 | 0 |
----------|---------|----------|---------|---------|-------------------
exit code: 0
```
2. The same run with `--reporter=text-summary`:
```
Statements : Unknown% ( 0/0 )
Branches : Unknown% ( 0/0 )
Functions : Unknown% ( 0/0 )
Lines : Unknown% ( 0/0 )
```
3. Control, so the threshold is visibly working when there is something to measure. `lib.js` holds one function, `t.js` is `require("./lib.js")`:
```
$ npx c8 --check-coverage --lines 100 node t.js
All files | 66.66 | 100 | 0 | 66.66 |
ERROR: Coverage for lines (66.66%) does not meet global threshold (100%)
exit code: 1
```
Three more shapes of the same thing, all exit 0:
* `npx c8 --check-coverage --lines 100 --include 'src/**' node t.js`, with the same real test as the control and a glob that matches none of it. The table prints 0 in every column and the gate passes.
* `npx c8 --check-coverage --per-file --lines 100 node -e "1"`.
* `npx c8 check-coverage --temp-directory ./empty-dir --lines 100`, which prints nothing at all and exits 0.
### Where it comes from
Two independent paths reach the same result, in case that is useful:
* `checkCoverage()` in `lib/commands/check-coverage.js` does `summary[key].pct < thresholds[key]`. For an empty map, istanbul's `CoverageSummary` sets `pct: 'Unknown'` (`istanbul-lib-coverage/lib/coverage-summary.js`), and `'Unknown' < 100` is `false`, so nothing is ever below a threshold.
* With `--per-file`, `map.files()` is empty, so the `forEach` body never runs and there is nothing to fail.
### What I am not proposing
I am not saying istanbul's `Unknown` is wrong. A file with no branches should not fail a branch threshold, and that is the same mechanism doing something useful. I also have no view I would defend on which way c8 should go: fail on an empty coverage map by default, warn, put it behind a flag, or document that `check-coverage` means "no measured metric is below the threshold". Each of those changes what existing CI does, including for people who run c8 over a process that legitimately produces no JS coverage, so the choice looks like yours rather than mine. No PR from me. If you land on a direction and want it written up, I am glad to do that.
Of the shapes above, the `--include` one seems most likely to happen by accident. A glob that stops matching after a directory is renamed leaves a coverage gate that always passes, and nothing in the output says it stopped measuring.
Contributor guide
Research direction
Start with lib/commands/check-coverage.js and istanbul-lib-coverage/lib/coverage-summary.js, then run the listed empty-coverage, unmatched --include, --per-file, and empty temp-directory reproductions. Compare the global and per-file paths and inspect existing check-coverage tests if present. Done requires an agreed behavior for empty coverage data, implemented consistently across these shapes, with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100