karma-runner / karma-runner/karma

socket `reconnect` from ping timeout does not work correctly

Open
#3,652 14 comments 0 reactions 1 assignee Claimed by @devoto13 View on GitHub
Dominant language
JavaScript
Stars
12k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

This is a long standing issue I've tried to fix in the past.

A user in my company hit a case that failed 30% of the time or so and he ran down the reason:

- the client takes a long hunk of time (parsing or running a test)
- the server hits the pingTimeout, disconnects, waits for reconnect
- the client sends results << These results are lost
- the client gets pingTimeout, tries reconnect (unsure)
- server sees reconnect as a new socket and connects.
- server never sees results, unpredictable result.

As far as I can tell, any data that the client sends after the server thinks the socket is disconnected is lost. The socket.io layer has no replay and neither does karma.

The problem is fundamental. Either we implement replay or we work to avoid disconnect but fail if we hit it.

Replay is not super hard since we know that tests are finite in duration.

But I think that, for the most part, disconnect is self-inflicted. The client is parsing a huge chunk of JS or it is off doing a long test: it fails to respond to a ping. The browser is there and the runtime is live, but we are starving the ping reply. If we increase the pingTimeout we can avoid the disconnect.

The other ways that the pingTimeout can be hit is browser/tab crash or network partition. With a large pingTimeout we wait a long time to discover the tab crash.

I can't come up with a scenario where automatic socket reconnect makes sense for us. Any time we lose connection, we lose data and the test is invalid. We could say that, eg logging up until the test starts is "optional" so we could reconnect if we lost connection before start. But if we lose connection before start the thing we need to figure out why is exactly those logs.

I think we should flush the entire reconnect logic, all of the timeouts other than pingTimeout, and retry logic in favor error messages directing users to look into the issue and then increase the pingTimeout.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.