api: Lock client doesn't use jitter while waiting to acquire a lock
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
When a lock client can't acquire a lock due to leader not being elected of a lock timeout, it uses a fixed 5 second delay.
https://github.com/hashicorp/consul/blob/cdcfd0e4f95afd9f6d868b0e2fbed3044b41975e/api/lock.go#L243
In cases where leadership is lost in a cluster where there are many clients contending on locks, those clients all get in sync and attempt to claim locks in a thundering herd every 5 seconds afterwards.
This can cause a single leader election to turn into a more complex prolonged outage due to the unusual spikes of writes.
Everywhere else in Consul we use Jitter liberally to prevent this. We should review the lock library to make sure all waits during the lock acquisition loop have jitter added to prevent this synchronisation.
Contributor guide
Assessment
This issue has not been assessed yet.