hashicorp / hashicorp/consul

Need a better way to handle lock acquisition failure

Open
#2,767 2 comments 0 reactions 0 assignees View on GitHub
theme/api type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

Please see #1008 (https://github.com/hashicorp/consul/pull/1008#issuecomment-110226523)

>The new behavior is:
1. 3 agents would contend for the lock.
2. 1 would get it.
3. 2, 3 get an false return from kv.Acquire
4. 2, 3 immediately do a non-blocking read on the key.
5. 2, 3 see a session ID on the key, jump back to a blocking read (WAIT).
6. After 1 second, 1 unlocks the key.
7. 2, 3 contend for the key immediately since the blocking read returned.
8. 2 acquires it, 3 gets a false return from kv.Acquire.
9. 3 immediately does a non-blocking read on the key.
10. 3 sees a session ID on the key and jumps back to a blocking read (WAIT).
11. 2 does its business, releases the lock after 1 second.
12. 3 contends for and acquires the lock due to the blocking read returning immediately.
Total time passed: 2 seconds.

Let's assume that the first agent releases the lock somewhere between 3 and 4. Then agent 2 and 3 see a blank session on the key which leads to make them [sleep 5 seconds to avoid a hot-loop](https://github.com/hashicorp/consul/blob/master/api/lock.go#L226).

Would it be better to try to acquire a lock again instead of a long sleep for nothing? Or it would be much better if a server could provide additional info of key which is in a lock-delay state. A blank session seems too vague.

Contributor guide

Open the contributing guide

Research direction

Read api/lock.go around line 226 and the discussion linked from PR #1008. Reproduce the described contention and lock-delay timing, then determine the intended behavior for a blank session and whether retrying or exposing additional server state is required. Done means the lock path avoids unnecessary delays without introducing a hot loop or breaking lock-delay semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.