[CURATOR-678] InterProcessMutex#release caused inconsistency between zk node and local cache if encountering zk connection lost

We experienced a problem that

an InterProcessMutex participant acquired the lock -> when release() was running, it encountered zk connection lost, then there was inconsistency as in codes https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/locks/InterProcessMutex.java#L139

to line 143, that the zk node deletion threw exception for connection lost, but the local cached `threadData` still removed it.

As a result, even when the zk connection recovered later, ALL following acquire() failed due to the inconsistency (not present in local `threadData` but the OLD zk node were still present).

Please help confirm this behavior. I think it is bug and curator should fix the inconsistency, a suggestion is to remove the local data ONLY after znode deletion is a success. Also, the same problematic code seems appearing in many other similar recipes such as `InterProcessSemaphore`.

Stacktrace:

```

Failed to release mutex for xxxxxxxxxxxxx org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /xxxx/_c_65fb02ef-9b1d-4c8c-b715-5c97f82ae0d3-lock-0000000000 at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:2001) ~[zookeeper-3.6.3.jar:3.6.3] at org.apache.curator.framework.imps.DeleteBuilderImpl$6.call(DeleteBuilderImpl.java:313) ~[curator-framework-5.3.0.jar:5.3.0] at org.apache.curator.framework.imps.DeleteBuilderImpl$6.call(DeleteBuilderImpl.java:301) ~[curator-framework-5.3.0.jar:5.3.0] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93) ~[curator-client-5.3.0.jar:?] at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:298) ~[curator-framework-5.3.0.jar:5.3.0] at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:282) ~[curator-framework-5.3.0.jar:5.3.0] at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35) ~[curator-framework-5.3.0.jar:5.3.0] at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:347) ~[curator-recipes-5.3.0.jar:5.3.0] at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:124) ~[curator-recipes-5.3.0.jar:5.3.0] at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154) ~[curator-recipes-5.3.0.jar:5.3.0] at 

... ...

```

---
Originally reported by rikimberley, imported from: InterProcessMutex#release caused inconsistency between zk node and local cache if encountering zk connection lost

• assignee: eolivelli
  
• status: Open
  
• priority: Major
  
• resolution: Unresolved
  
• imported: 2025-01-21