apache / apache/pulsar

Flaky-test: PulsarWorkerRebalanceDrainTest.testRebalanceWorkers. Function worker leader changes unpredictably causing flakiness.

Open
#24,327 1 comment 0 reactions 0 assignees View on GitHub
area/test type/flaky-tests
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Example failure

https://gist.github.com/ShivrajJ/134d23d79a6e122677fd5b300c4de3fa

### Exception stacktrace

I added a debug-level log with the topic stats, so line numbers in the stack trace might be offset a little, see https://github.com/cognitree/pulsar/pull/22 for the changes

```text
Expected :true
Actual :false

java.lang.AssertionError:
at org.testng.Assert.fail(Assert.java:110)
at org.testng.Assert.failNotEquals(Assert.java:1577)
at org.testng.Assert.assertTrue(Assert.java:56)
at org.testng.Assert.assertTrue(Assert.java:66)
at org.apache.pulsar.tests.integration.functions.java.PulsarWorkerRebalanceDrainTest.testRebalance(PulsarWorkerRebalanceDrainTest.java:350)
at org.apache.pulsar.tests.integration.functions.java.PulsarWorkerRebalanceDrainTest.testRebalanceWorkers(PulsarWorkerRebalanceDrainTest.java:70)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:65)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)
```

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by locating PulsarWorkerRebalanceDrainTest.java and reading testRebalanceWorkers and testRebalance around lines 70 and 350. Run the integration test and compare its behavior with the linked failure details and topic-stat logs. Done means the rebalance test no longer fails because the worker leader changes unpredictably.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.