apache / apache/pulsar

Flaky-test: SaslAuthenticateTest.testSaslOnlyAuthFirstStage

Open
#24,337 1 comment 0 reactions 0 assignees View on GitHub
area/test type/flaky-tests
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Example failure

https://gist.github.com/storm-08/61ff3104a1f0492cfa98c600708faa3e

### Exception stacktrace
Please note I have added Thread.sleep(5) in my test case so line number may offset by 1 in trace.
```text
java.lang.AssertionError:
Expected :10
Actual :7

at org.testng.Assert.fail(Assert.java:110)
at org.testng.Assert.failNotEquals(Assert.java:1577)
at org.testng.Assert.assertEqualsImpl(Assert.java:149)
at org.testng.Assert.assertEquals(Assert.java:131)
at org.testng.Assert.assertEquals(Assert.java:1418)
at org.testng.Assert.assertEquals(Assert.java:1382)
at org.testng.Assert.assertEquals(Assert.java:1428)
at org.apache.pulsar.broker.authentication.SaslAuthenticateTest.testSaslOnlyAuthFirstStage(SaslAuthenticateTest.java:332)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:829)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
at org.testng.SuiteRunner.run(SuiteRunner.java:330)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
at org.testng.TestNG.runSuites(TestNG.java:1099)
at org.testng.TestNG.run(TestNG.java:1067)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:65)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)
```

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

### Description
testSaslOnlyAuthFirstStage can intermittently fail due to a strict cache size assertion affected by entry expiry timing.

#### Expected Behaviour
The test should reliably verify that some entries exist in the cache without assuming exact timing.

#### Actual Behaviour
Test sometimes fails when entries from the first loop expire before the size assertion, leading to less than 10 in the cache.

#### Steps to reproduce
Simply add Thread.sleep(5) before the end of first for loop and then run the test case.

### Notes
- The expiry time (inflightSaslContextExpiryMs) is set to 50ms.
- The first loop can take >50ms, causing early entries to be evicted before the assertion.
- The strict assertEquals(cache.size(), 10) made the test flaky.

Please see [this](https://github.com/cognitree/pulsar/pull/23) for the changes.

Link to detailed logs and trace of error - https://gist.github.com/storm-08/61ff3104a1f0492cfa98c600708faa3e

Please note I have added Thread.sleep(5) in my test case so line number may offset by 1 in trace.

Contributor guide

Open the contributing guide

Research direction

Start with SaslAuthenticateTest.java and the testSaslOnlyAuthFirstStage method named in the stack trace. Run that test, then reproduce the timing described by adding the 5 ms sleep before the first loop ends. Done means the test remains reliable when entries expire during setup and no longer depends on the exact cache size of 10.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
authentication, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.