Flaky test in MultiNodesOfflineClusterIntegrationTest
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
`MultiNodesOfflineClusterIntegrationTest.testServerHardFailure` shuts down one server and then executes a request, expecting it to fail with `Connection refused`. But the shut down call is not blocking. Specifically, it doesn't wait until the server is stopped. Therefore there is a race condition between the shut down call and the request.
In case the request is processed by the broker once the server is stopped, then the request fails with error code `BROKER_REQUEST_SEND_ERROR_CODE` and the expected `Connection refused` message. But it can also happen that the request is started to be executed before the broker knows that the server is down, in which case the request can theoretically success or fail with other error messages.
In my executions I wasn't able to make the request to success, but from time to time the request fails with `BROKER_REQUEST_SEND_ERROR_CODE` error code (as expected) but `Connection reset` message. I think I found other messages as well, but I'm not 100% sure. That makes the test fail when in my opinion it shouldn't.
I can see two solutions: The simple and empirically good enough is to expect the error code but do not assert on the message. The other solution would be to make the shut down call blocking. That would make the test 100% repeatable, but it wouldn't cover all the cases (as we would never check what happens when the server is killed in the middle of a call).
This issue was discovered while working on #10528, which includes the empirical solution described above.
Contributor guide
Research direction
Start with MultiNodesOfflineClusterIntegrationTest.testServerHardFailure and inspect the shutdown and request assertions. Review the approach described in #10528, then run the integration test to verify it handles the shutdown race consistently while preserving the expected broker error code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100