Flaky-test: ZkSessionExpireTest.cleanup hangs
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Example failure
https://github.com/apache/pulsar/actions/runs/11125404278/job/30927545059?pr=23327#step:13:107
### Exception stacktrace
ZkSessionExpireTest.cleanup hangs in `ModularLoadManagerImpl.disableBroker`
```
"main" #1 [2383] prio=5 os_prio=0 cpu=101566.56ms elapsed=3520.41s tid=0x00007f3618030a30 nid=2383 waiting on condition [0x00007f3621f54000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.4/Native Method)
- parking to wait for <0x0000100041604dc8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.4/LockSupport.java:221)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.4/CompletableFuture.java:1864)
at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.4/ForkJoinPool.java:3780)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.4/ForkJoinPool.java:3725)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@21.0.4/CompletableFuture.java:1898)
at java.util.concurrent.CompletableFuture.join(java.base@21.0.4/CompletableFuture.java:2117)
at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.disableBroker(ModularLoadManagerImpl.java:603)
at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.disableBroker(ModularLoadManagerWrapper.java:47)
at org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully(BrokerService.java:949)
at org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully(BrokerService.java:936)
at org.apache.pulsar.broker.PulsarService.closeAsync(PulsarService.java:520)
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:479)
at org.apache.pulsar.broker.service.NetworkErrorTestBase.cleanup(NetworkErrorTestBase.java:215)
at org.apache.pulsar.broker.service.ZkSessionExpireTest.cleanup(ZkSessionExpireTest.java:50)
```
Thread dump: https://gist.github.com/lhotari/68578ba1354d4833af489b9ac3658c8c
Analysed in jstack.review: https://jstack.review/?https://gist.github.com/lhotari/68578ba1354d4833af489b9ac3658c8c#tda_1_dump
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start by reading ZkSessionExpireTest.java cleanup and NetworkErrorTestBase.java cleanup, then inspect the disableBroker call at ModularLoadManagerImpl.java:603. Reproduce the flaky test and examine the supplied stack trace and thread dump. Done means cleanup completes reliably without hanging in disableBroker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100