apache / apache/pulsar

Thread deadlock in stress test

Open
#14,443 6 comments 0 reactions 0 assignees View on GitHub
help wanted lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: [e.g. iOS]

**Additional context**
"pulsar-timer-66-1" #2504 prio=5 os_prio=0 tid=0x00007ff18c194000 nid=0xaa3 waiting for monitor entry [0x00007ff1cc1e8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.pulsar.client.impl.ProducerImpl.run(ProducerImpl.java:1552)
- waiting to lock <0x00000000c8032000> (a org.apache.pulsar.client.impl.ProducerImpl)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

pulsar-timer-66-1" #2504 prio=5 os_prio=0 tid=0x00007ff18c194000 nid=0xaa3 waiting for monitor entry [0x00007ff1cc1e8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.pulsar.client.impl.ProducerImpl.run(ProducerImpl.java:1552)
- waiting to lock <0x00000000c8032000> (a org.apache.pulsar.client.impl.ProducerImpl)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

"pulsar-client-io-63-4" #2507 prio=5 os_prio=0 tid=0x00007ff18c0e0000 nid=0xaa2 waiting on condition [0x00007ff1cc2e8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c8032140> (a java.util.concurrent.Semaphore$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at org.apache.pulsar.client.impl.ProducerImpl.canEnqueueRequest(ProducerImpl.java:766)
at org.apache.pulsar.client.impl.ProducerImpl.sendAsync(ProducerImpl.java:377)
at org.apache.pulsar.client.impl.ProducerImpl.internalSendAsync(ProducerImpl.java:281)
at org.apache.pulsar.client.impl.ProducerBase.sendAsync(ProducerBase.java:76)
at com.chinamobile.tuxedo.amqp.store.OpAddMessage.initiate(OpAddMessage.java:129)
at com.chinamobile.tuxedo.amqp.store.MessageLogPulsarImpl.updateMessageLogOpened(MessageLogPulsarImpl.java:112)
at com.chinamobile.tuxedo.amqp.store.MessageLogPulsarImpl.lambda$openMessageLog$0(MessageLogPulsarImpl.java:93)
at com.chinamobile.tuxedo.amqp.store.MessageLogPulsarImpl$$Lambda$743/601077192.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.apache.pulsar.client.impl.ProducerImpl.lambda$resendMessages$19(ProducerImpl.java:1468)
- locked <0x00000000c8032000> (a org.apache.pulsar.client.impl.ProducerImpl)
at org.apache.pulsar.client.impl.ProducerImpl$$Lambda$765/1246310252.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

[https://blog.fastthread.io/2016/03/07/thread-dump-analysis-pattern-ripple-effect/](url)

Contributor guide

Open the contributing guide

Research direction

Start with the thread dump and the referenced ProducerImpl.java locations, especially lines 1552, 1468, 766, 377, and 281, to trace the monitor and semaphore interaction. Reproduce the reported stress-test deadlock if possible; done means the blocked ProducerImpl path no longer deadlocks under that workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.