karma-runner / karma-runner/karma

Different Coverage Numbers Based On Test Run Order

Open
#3,899 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
12k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

I have some tests that are producing different coverage numbers based on the order in which the tests are run.

I have components `A` and `B`; `B` is a dependency of `A`. If I run the tests for `A` then the tests for `B`, I get the coverage number for `A` that I expect but not for `B`; I get a much lower coverage number for `B`. However, if I reverse the order I get the order I get the coverage numbers I expect for `A` and `B`.

I believe this is because `A` uses some of the code in `B` so if it runs first the coverage for `B` is based on how `A` is using it. When the tests for `B` is run the coverage has already been calculated. This makes to me and matches what I am seeing. I am just wondering if this behavior is correct.

Sorry for the lack of actual code but I can't post it for proprietary reasons. I hope my descriptions makes sense. Thanks.

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or entry points are identified. Start by reproducing the coverage difference with the described A/B test-run orders, then trace how coverage is collected across dependent components. Done means determining whether order-dependent coverage is expected and documenting or fixing the behavior based on that finding.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.