hazelcast / hazelcast/hazelcast-cpp-client

[TRACKING ISSUE] Remove the deadlock between clientStateMutex and clusterViewMutex

Open
#798 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
91
Forks
54
Avg merge
1d 12m
Merged PRs (30d)
5

Description

The tracking issue for the Java side PR.

**Remove the deadlock between clientStateMutex and clusterViewMutex**

See https://github.com/hazelcast/hazelcast/pull/18033 for details.

---

We have captured a deadlock in a failing test:
```
java.lang.Thread.State: BLOCKED, on lock=java.lang.Object@467064ec, owned by pool-1004-thread-1, id=100868, cpu=3827564 nsecs, usr=0 nsecs, blocked=121349 msecs, waited=18 msecs
at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:711)
at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:190)
at com.hazelcast.client.impl.spi.impl.ClientClusterServiceImpl.detectMembershipEvents(ClientClusterServiceImpl.java:280)
at com.hazelcast.client.impl.spi.impl.ClientClusterServiceImpl.handleMembersViewEvent(ClientClusterServiceImpl.java:338)
at com.hazelcast.client.impl.spi.impl.listener.ClientClusterViewListenerService$ClusterViewListenerHandler.handleMembersViewEvent(ClientClusterViewListenerService.java:90)
at com.hazelcast.client.impl.protocol.codec.ClientAddClusterViewListenerCodec$AbstractEventHandler.handle(ClientAddClusterViewListenerCodec.java:116)
at com.hazelcast.client.impl.spi.impl.listener.ClientClusterViewListenerService$ClusterViewListenerHandler.handle(ClientClusterViewListenerService.java:64)
at com.hazelcast.client.impl.spi.impl.listener.ClientListenerServiceImpl.handleEventMessageOnCallingThread(ClientListenerServiceImpl.java:189)
at com.hazelcast.client.impl.spi.impl.listener.ClientListenerServiceImpl.lambda$handleEventMessage$2(ClientListenerServiceImpl.java:167)
at com.hazelcast.client.impl.spi.impl.listener.ClientListenerServiceImpl$$Lambda$1821/1960977249.run(Unknown Source)
at com.hazelcast.internal.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:245)
at com.hazelcast.internal.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:228)

java.lang.Thread.State: BLOCKED, on lock=java.lang.Object@35eabfb3, owned by hz.client_313.event-7092, id=100854, cpu=3544732 nsecs, usr=0 nsecs, blocked=121349 msecs, waited=34 msecs
at com.hazelcast.client.impl.spi.impl.ClientClusterServiceImpl.clearMemberListVersion(ClientClusterServiceImpl.java:203)
at com.hazelcast.client.impl.spi.impl.listener.ClientClusterViewListenerService$ClusterViewListenerHandler.beforeListenerRegister(ClientClusterViewListenerService.java:75)
at com.hazelcast.client.impl.spi.impl.listener.ClientClusterViewListenerService.tryRegister(ClientClusterViewListenerService.java:129)
at com.hazelcast.client.impl.spi.impl.listener.ClientClusterViewListenerService.tryReregisterToRandomConnection(ClientClusterViewListenerService.java:116)
at com.hazelcast.client.impl.spi.impl.listener.ClientClusterViewListenerService.connectionRemoved(ClientClusterViewListenerService.java:106)
at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.fireConnectionRemovedEvent(TcpClientConnectionManager.java:600)
at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:722)
at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:190)
at com.hazelcast.client.test.TestClientRegistry$MockedTcpClientConnection$4.run(TestClientRegistry.java:316)
at com.hazelcast.client.test.TwoWayBlockableExecutor$BlockableRunnable.run(TwoWayBlockableExecutor.java:98)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
These two threads are waiting for each other because they are trying to lock same mutexes in reverse order.

In the first stacktrace, a member REMOVED event could cause closing of a member under `clusterViewLock` which will require `clientStateMutex`
Fix: The responsibility of closing of the connection is taken from ClientCLusterService and given to TcpClientConnectioManager.

In the second stacktrace, Someone closing the connection could potentially cause a reregistration listener under `clientStateMutex` which will requre `clusterViewLock`
Fix: fireConenctionRemovedEvent is offloaded to an executor so that it will not held the mutex when fired.

fixes https://github.com/hazelcast/hazelcast/issues/18032

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.