line / line/centraldogma

Dead lock while shutting down ZooKeeper replica

Open
#599 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

defect
Dominant language
Java
Stars
668
Forks
135
Avg merge
4d 19h
Merged PRs (30d)
7

Description

https://github.com/line/centraldogma/pull/592/checks?check_run_id=2636423174

There are 3 points threads are stuck at.

"Test worker" #12 prio=5 os_prio=31 cpu=10611.68ms elapsed=3088.79s tid=0x00007f9544541000 nid=0x9e03 waiting on condition  [0x0000700010d46000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)
	- parking to wait for  <0x00000007ec68e120> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.11/LockSupport.java:194)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.11/CompletableFuture.java:1796)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.11/ForkJoinPool.java:3128)
	at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.11/CompletableFuture.java:1823)
	at java.util.concurrent.CompletableFuture.join(java.base@11.0.11/CompletableFuture.java:2043)
	at com.linecorp.centraldogma.server.internal.replication.Cluster.close(Cluster.java:53)
	at com.linecorp.centraldogma.server.internal.replication.ZooKeeperCommandExecutorTest.testRace(ZooKeeperCommandExecutorTest.java:185)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.11/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.11/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)

"zookeeper-leader-selector-305-1" #2920 daemon prio=5 os_prio=31 cpu=220.65ms elapsed=2844.22s tid=0x00007f95478d6000 nid=0x75c0f in Object.wait()  [0x0000700021379000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.11/Native Method)
	- waiting on <no object reference available>
	at java.lang.Object.wait(java.base@11.0.11/Object.java:328)
	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529)
	- waiting to re-lock in wait() <0x00000007f45d7ee8> (a org.apache.zookeeper.ClientCnxn$Packet)
	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2705)
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:242)
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:231)
	at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:67)
	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:81)
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:228)
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:219)
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:41)
	at org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:154)
	at org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:196)
	at org.apache.curator.framework.recipes.locks.LockInternals.internalLockLoop(LockInternals.java:284)
	at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:226)
	at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:237)
	at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:89)
	at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:412)
	at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:483)
	at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:66)
	at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:247)
	at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:241)
	at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
	at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.11/Executors.java:515)
	at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.11/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.11/ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)

"zookeeper-leader-selector-317-1" #2942 daemon prio=5 os_prio=31 cpu=1.54ms elapsed=2844.21s tid=0x00007f954767a000 nid=0x26013 in Object.wait()  [0x00007000227b6000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.11/Native Method)
	- waiting on <no object reference available>
	at java.lang.Object.wait(java.base@11.0.11/Object.java:328)
	at com.linecorp.centraldogma.server.internal.replication.ZooKeeperCommandExecutor$OldLogRemover.takeLeadership(ZooKeeperCommandExecutor.java:212)
	- waiting to re-lock in wait() <0x00000007ec7a9f80> (a com.linecorp.centraldogma.server.internal.replication.ZooKeeperCommandExecutor$OldLogRemover)
	at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:576)
	at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:425)
	at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:483)
	at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:66)
	at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:247)
	at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:241)
	at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
	at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.11/Executors.java:515)
	at java.util.concurrent.FutureTask.run(java.base@11.0.11/FutureTask.java:264)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.11/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.11/ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Cluster.close in Cluster.java and the failing testRace case in ZooKeeperCommandExecutorTest.java. Trace the shutdown interaction with ZooKeeperCommandExecutor$OldLogRemover.takeLeadership and the leader-selector wait states shown in the thread dumps. Reproduce the shutdown race and verify that the replica closes without leaving the test blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.