bcoe / bcoe/c8

Reaching mocha timeout leads to incorrect coverage (vscode extension testing)

Open
#416 0 comments 0 reactions 0 assignees View on GitHub
bug p2
Dominant language
JavaScript
Stars
2.1k
Forks
99
PR merge metrics
No merged PRs in 30d

Description

* **Version**: v18.7.0
* **Platform**: Linux x86_64 (should affect all)

This is more of a note rather than an bug report and would probably be more appropriate as a GitHub Discussion if they were enabled.

We just starting using `c8` for coverage in our VS Code extension https://github.com/fortran-lang/vscode-fortran-support and what I noticed was that if we hit our `timeout` in our `mocha` unittests `c8` carries on with the coverage report as if the unittests have run succesully. Maybe it would be beneficial if `c8` threw some kind of warning that the coverage **might** be inaccurate because you reached `timeout`.

In our case, we had some relatively expensive `suiteSetup` and `suiteTeardown`, so even though our tests were passing `c8` would produce very incorrect reports, e.g. 40% as opposed to 70%. It took me a while to figure out what was going on given the nightmare that vscode unittesting tends to be.

To reproduce you can checkout our repo on commit https://github.com/fortran-lang/vscode-fortran-support/pull/629/commits/801c40ce4a80af7452931efda1b3f15e2f271ade and

```sh
npm ci
npm run coverage
```

An example where coverage is very incorrect is the file `linter-provider.ts`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.