Thread deadlock in stress test
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
**Additional context**
"pulsar-timer-66-1" #2504 prio=5 os_prio=0 tid=0x00007ff18c194000 nid=0xaa3 waiting for monitor entry [0x00007ff1cc1e8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.pulsar.client.impl.ProducerImpl.run(ProducerImpl.java:1552)
- waiting to lock <0x00000000c8032000> (a org.apache.pulsar.client.impl.ProducerImpl)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
pulsar-timer-66-1" #2504 prio=5 os_prio=0 tid=0x00007ff18c194000 nid=0xaa3 waiting for monitor entry [0x00007ff1cc1e8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.pulsar.client.impl.ProducerImpl.run(ProducerImpl.java:1552)
- waiting to lock <0x00000000c8032000> (a org.apache.pulsar.client.impl.ProducerImpl)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"pulsar-client-io-63-4" #2507 prio=5 os_prio=0 tid=0x00007ff18c0e0000 nid=0xaa2 waiting on condition [0x00007ff1cc2e8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c8032140> (a java.util.concurrent.Semaphore$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at org.apache.pulsar.client.impl.ProducerImpl.canEnqueueRequest(ProducerImpl.java:766)
at org.apache.pulsar.client.impl.ProducerImpl.sendAsync(ProducerImpl.java:377)
at org.apache.pulsar.client.impl.ProducerImpl.internalSendAsync(ProducerImpl.java:281)
at org.apache.pulsar.client.impl.ProducerBase.sendAsync(ProducerBase.java:76)
at com.chinamobile.tuxedo.amqp.store.OpAddMessage.initiate(OpAddMessage.java:129)
at com.chinamobile.tuxedo.amqp.store.MessageLogPulsarImpl.updateMessageLogOpened(MessageLogPulsarImpl.java:112)
at com.chinamobile.tuxedo.amqp.store.MessageLogPulsarImpl.lambda$openMessageLog$0(MessageLogPulsarImpl.java:93)
at com.chinamobile.tuxedo.amqp.store.MessageLogPulsarImpl$$Lambda$743/601077192.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.apache.pulsar.client.impl.ProducerImpl.lambda$resendMessages$19(ProducerImpl.java:1468)
- locked <0x00000000c8032000> (a org.apache.pulsar.client.impl.ProducerImpl)
at org.apache.pulsar.client.impl.ProducerImpl$$Lambda$765/1246310252.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[https://blog.fastthread.io/2016/03/07/thread-dump-analysis-pattern-ripple-effect/](url)
Contributor guide
Research direction
Start with the thread dump and the referenced ProducerImpl.java locations, especially lines 1552, 1468, 766, 377, and 281, to trace the monitor and semaphore interaction. Reproduce the reported stress-test deadlock if possible; done means the blocked ProducerImpl path no longer deadlocks under that workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100