Flaky-test: SaslAuthenticateTest.testSaslOnlyAuthFirstStage
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Example failure
https://gist.github.com/storm-08/61ff3104a1f0492cfa98c600708faa3e
### Exception stacktrace
Please note I have added Thread.sleep(5) in my test case so line number may offset by 1 in trace.
```text
java.lang.AssertionError:
Expected :10
Actual :7
at org.testng.Assert.fail(Assert.java:110)
at org.testng.Assert.failNotEquals(Assert.java:1577)
at org.testng.Assert.assertEqualsImpl(Assert.java:149)
at org.testng.Assert.assertEquals(Assert.java:131)
at org.testng.Assert.assertEquals(Assert.java:1418)
at org.testng.Assert.assertEquals(Assert.java:1382)
at org.testng.Assert.assertEquals(Assert.java:1428)
at org.apache.pulsar.broker.authentication.SaslAuthenticateTest.testSaslOnlyAuthFirstStage(SaslAuthenticateTest.java:332)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:65)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)
```
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
### Description
testSaslOnlyAuthFirstStage can intermittently fail due to a strict cache size assertion affected by entry expiry timing.
#### Expected Behaviour
The test should reliably verify that some entries exist in the cache without assuming exact timing.
#### Actual Behaviour
Test sometimes fails when entries from the first loop expire before the size assertion, leading to less than 10 in the cache.
#### Steps to reproduce
Simply add Thread.sleep(5) before the end of first for loop and then run the test case.
### Notes
- The expiry time (inflightSaslContextExpiryMs) is set to 50ms.
- The first loop can take >50ms, causing early entries to be evicted before the assertion.
- The strict assertEquals(cache.size(), 10) made the test flaky.
Please see [this](https://github.com/cognitree/pulsar/pull/23) for the changes.
Link to detailed logs and trace of error - https://gist.github.com/storm-08/61ff3104a1f0492cfa98c600708faa3e
Please note I have added Thread.sleep(5) in my test case so line number may offset by 1 in trace.
Contributor guide
Research direction
Start with SaslAuthenticateTest.java and the testSaslOnlyAuthFirstStage method named in the stack trace. Run that test, then reproduce the timing described by adding the 5 ms sleep before the first loop ends. Done means the test remains reliable when entries expire during setup and no longer depends on the exact cache size of 10.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- authentication, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100