Flaky-test: ExtensibleLoadManagerImplBaseTest.cleanup hangs in TableViewLoadDataStoreImpl.closeTableView due to a deadlock
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Example failure
https://github.com/apache/pulsar/actions/runs/11368523712/job/31653386805?pr=23468#step:10:616
thread dump: https://gist.github.com/lhotari/17557838cea2e4d4f4f1556fd4caec98
jstack.review analysis: https://jstack.review/?https://gist.github.com/lhotari/17557838cea2e4d4f4f1556fd4caec98#tda_1_dump
### Exception stacktrace
```
"main" #1 prio=5 os_prio=0 cpu=67851.86ms elapsed=5366.03s tid=0x00007f52bc02f210 nid=0x33a0 waiting on condition [0x00007f52c21fb000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.13/Native Method)
- parking to wait for <0x00001000319052f8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.13/LockSupport.java:211)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.13/CompletableFuture.java:1864)
at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.13/ForkJoinPool.java:3465)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.13/ForkJoinPool.java:3436)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.13/CompletableFuture.java:1898)
at java.util.concurrent.CompletableFuture.get(java.base@17.0.13/CompletableFuture.java:2072)
at org.apache.pulsar.client.impl.TableViewImpl.close(TableViewImpl.java:200)
at org.apache.pulsar.broker.loadbalance.extensions.store.TableViewLoadDataStoreImpl.closeTableView(TableViewLoadDataStoreImpl.java:148)
- locked <0x000010002025ce08> (a org.apache.pulsar.broker.loadbalance.extensions.store.TableViewLoadDataStoreImpl)
at org.apache.pulsar.broker.loadbalance.extensions.store.TableViewLoadDataStoreImpl.close(TableViewLoadDataStoreImpl.java:204)
- locked <0x000010002025ce08> (a org.apache.pulsar.broker.loadbalance.extensions.store.TableViewLoadDataStoreImpl)
at org.apache.pulsar.broker.loadbalance.extensions.store.TableViewLoadDataStoreImpl.shutdown(TableViewLoadDataStoreImpl.java:209)
- locked <0x000010002025ce08> (a org.apache.pulsar.broker.loadbalance.extensions.store.TableViewLoadDataStoreImpl)
at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl.stopLoadDataReportTasks(ExtensibleLoadManagerImpl.java:816)
at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl.disableBroker(ExtensibleLoadManagerImpl.java:1036)
at java.lang.invoke.LambdaForm$DMH/0x00007f5038410c00.invokeVirtual(java.base@17.0.13/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038540c00.invoke(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038448400.invoke(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038448800.invokeExact_MT(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.MethodHandle.invokeWithArguments(java.base@17.0.13/MethodHandle.java:732)
at org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$mHxiQzAK.invokeWithArguments(Unknown Source)
at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.invoke(InstrumentationMemberAccessor.java:251)
at org.mockito.internal.util.reflection.ModuleMemberAccessor.invoke(ModuleMemberAccessor.java:55)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.tryInvoke(MockMethodAdvice.java:314)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice$RealMethodCall.invoke(MockMethodAdvice.java:234)
at org.mockito.internal.invocation.InterceptedInvocation.callRealMethod(InterceptedInvocation.java:142)
at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:45)
at org.mockito.Answers.answer(Answers.java:90)
at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:111)
at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl.disableBroker(ExtensibleLoadManagerImpl.java:1033)
at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerWrapper.disableBroker(ExtensibleLoadManagerWrapper.java:88)
at org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully(BrokerService.java:975)
at org.apache.pulsar.broker.service.BrokerService.unloadNamespaceBundlesGracefully(BrokerService.java:962)
at org.apache.pulsar.broker.PulsarService.closeAsync(PulsarService.java:525)
at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@17.0.13/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00007f5038448000.invoke(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038448400.invoke(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038448800.invokeExact_MT(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.MethodHandle.invokeWithArguments(java.base@17.0.13/MethodHandle.java:732)
at org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$mHxiQzAK.invokeWithArguments(Unknown Source)
at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.invoke(InstrumentationMemberAccessor.java:251)
at org.mockito.internal.util.reflection.ModuleMemberAccessor.invoke(ModuleMemberAccessor.java:55)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.tryInvoke(MockMethodAdvice.java:314)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice$RealMethodCall.invoke(MockMethodAdvice.java:234)
at org.mockito.internal.invocation.InterceptedInvocation.callRealMethod(InterceptedInvocation.java:142)
at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:45)
at org.mockito.Answers.answer(Answers.java:90)
at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:111)
at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
at org.apache.pulsar.broker.PulsarService.closeAsync(PulsarService.java:509)
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:484)
at java.lang.invoke.LambdaForm$DMH/0x00007f5038410c00.invokeVirtual(java.base@17.0.13/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038540c00.invoke(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038448400.invoke(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f5038448800.invokeExact_MT(java.base@17.0.13/LambdaForm$MH)
at java.lang.invoke.MethodHandle.invokeWithArguments(java.base@17.0.13/MethodHandle.java:732)
at org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$mHxiQzAK.invokeWithArguments(Unknown Source)
at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.invoke(InstrumentationMemberAccessor.java:251)
at org.mockito.internal.util.reflection.ModuleMemberAccessor.invoke(ModuleMemberAccessor.java:55)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.tryInvoke(MockMethodAdvice.java:314)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice$RealMethodCall.invoke(MockMethodAdvice.java:234)
at org.mockito.internal.invocation.InterceptedInvocation.callRealMethod(InterceptedInvocation.java:142)
at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:45)
at org.mockito.Answers.answer(Answers.java:90)
at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:111)
at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:484)
at org.apache.pulsar.broker.testcontext.PulsarTestContext$StartableCustomBuilder.lambda$initializePulsarServices$0(PulsarTestContext.java:778)
at org.apache.pulsar.broker.testcontext.PulsarTestContext$StartableCustomBuilder$$Lambda$568/0x00007f503853d598.close(Unknown Source)
at org.apache.pulsar.broker.testcontext.PulsarTestContext.callCloseables(PulsarTestContext.java:218)
at org.apache.pulsar.broker.testcontext.PulsarTestContext.close(PulsarTestContext.java:212)
at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.internalCleanup(MockedPulsarServiceBaseTest.java:287)
at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplBaseTest.cleanup(ExtensibleLoadManagerImplBaseTest.java:171)
```
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with ExtensibleLoadManagerImplBaseTest.cleanup and trace the shutdown path through MockedPulsarServiceBaseTest, PulsarTestContext, ExtensibleLoadManagerImpl, and TableViewLoadDataStoreImpl.closeTableView. Reproduce the flaky test and use the supplied thread dump to investigate the interaction with TableViewImpl.close; done means cleanup completes without hanging or deadlocking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100