Need a better way to handle lock acquisition failure
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
Please see #1008 (https://github.com/hashicorp/consul/pull/1008#issuecomment-110226523)
>The new behavior is:
1. 3 agents would contend for the lock.
2. 1 would get it.
3. 2, 3 get an false return from kv.Acquire
4. 2, 3 immediately do a non-blocking read on the key.
5. 2, 3 see a session ID on the key, jump back to a blocking read (WAIT).
6. After 1 second, 1 unlocks the key.
7. 2, 3 contend for the key immediately since the blocking read returned.
8. 2 acquires it, 3 gets a false return from kv.Acquire.
9. 3 immediately does a non-blocking read on the key.
10. 3 sees a session ID on the key and jumps back to a blocking read (WAIT).
11. 2 does its business, releases the lock after 1 second.
12. 3 contends for and acquires the lock due to the blocking read returning immediately.
Total time passed: 2 seconds.
Let's assume that the first agent releases the lock somewhere between 3 and 4. Then agent 2 and 3 see a blank session on the key which leads to make them [sleep 5 seconds to avoid a hot-loop](https://github.com/hashicorp/consul/blob/master/api/lock.go#L226).
Would it be better to try to acquire a lock again instead of a long sleep for nothing? Or it would be much better if a server could provide additional info of key which is in a lock-delay state. A blank session seems too vague.
Contributor guide
Research direction
Read api/lock.go around line 226 and the discussion linked from PR #1008. Reproduce the described contention and lock-delay timing, then determine the intended behavior for a blank session and whether retrying or exposing additional server state is required. Done means the lock path avoids unnecessary delays without introducing a hot loop or breaking lock-delay semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100