apache / apache/pulsar

Flaky-test: ZkSessionExpireTest.cleanup hangs

Open
#23,388 2 comments 0 reactions 0 assignees View on GitHub
area/test type/flaky-tests
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Example failure

https://github.com/apache/pulsar/actions/runs/11125404278/job/30927545059?pr=23327#step:13:107

### Exception stacktrace

ZkSessionExpireTest.cleanup hangs in `ModularLoadManagerImpl.disableBroker`

```
"main" #1 [2383] prio=5 os_prio=0 cpu=101566.56ms elapsed=3520.41s tid=0x00007f3618030a30 nid=2383 waiting on condition [0x00007f3621f54000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.4/Native Method)
- parking to wait for <0x0000100041604dc8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.4/LockSupport.java:221)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.4/CompletableFuture.java:1864)
at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.4/ForkJoinPool.java:3780)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.4/ForkJoinPool.java:3725)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@21.0.4/CompletableFuture.java:1898)
at java.util.concurrent.CompletableFuture.join(java.base@21.0.4/CompletableFuture.java:2117)
at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.disableBroker(ModularLoadManagerImpl.java:603)
at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.disableBroker(ModularLoadManagerWrapper.java:47)
at org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully(BrokerService.java:949)
at org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully(BrokerService.java:936)
at org.apache.pulsar.broker.PulsarService.closeAsync(PulsarService.java:520)
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:479)
at org.apache.pulsar.broker.service.NetworkErrorTestBase.cleanup(NetworkErrorTestBase.java:215)
at org.apache.pulsar.broker.service.ZkSessionExpireTest.cleanup(ZkSessionExpireTest.java:50)
```

Thread dump: https://gist.github.com/lhotari/68578ba1354d4833af489b9ac3658c8c
Analysed in jstack.review: https://jstack.review/?https://gist.github.com/lhotari/68578ba1354d4833af489b9ac3658c8c#tda_1_dump

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading ZkSessionExpireTest.java cleanup and NetworkErrorTestBase.java cleanup, then inspect the disableBroker call at ModularLoadManagerImpl.java:603. Reproduce the flaky test and examine the supplied stack trace and thread dump. Done means cleanup completes reliably without hanging in disableBroker.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.