karma-runner / karma-runner/karma

Using browserDisconnectTolerance invalidates test suite

Open
#3,077 4 comments 7 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
12k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Expected behaviour

When a test fails, karma says that the test failed.

### Actual behaviour

When a test fails and chrome disconnects then reconnects (which is allowed if the `browserDisconnectTolerance` is set to something greater than 0), karma will only report the results of the tests that have run since reconnecting, ignoring any test failures from before chrome disconnected.

### Environment Details

- Karma version (output of `karma --version`): 1.7.1
- Relevant part of your `karma.config.js` file:
```
browserDisconnectTolerance: 1
```

### Steps to reproduce the behaviour

1. Set `browserDisconnectTolerance` to anything greater than 0.
2. Run a test that fails before the browser disconnects, then after reconnecting run a test that passes
3. _Expected_: The build should fail and karma should report the test failure from before the disconnection
_Actual_: The build passes because karma only reports the results of tests run since the reconnection

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction described in karma.config.js using browserDisconnectTolerance greater than 0: run a failing test before Chrome disconnects and a passing test after it reconnects. Trace how Karma collects and reports results across the reconnect, then verify that the earlier failure still makes the build fail and is reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.