hashicorp / hashicorp/consul

api: Lock client doesn't use jitter while waiting to acquire a lock

Open
#9,469 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

When a lock client can't acquire a lock due to leader not being elected of a lock timeout, it uses a fixed 5 second delay.

https://github.com/hashicorp/consul/blob/cdcfd0e4f95afd9f6d868b0e2fbed3044b41975e/api/lock.go#L243

In cases where leadership is lost in a cluster where there are many clients contending on locks, those clients all get in sync and attempt to claim locks in a thundering herd every 5 seconds afterwards.

This can cause a single leader election to turn into a more complex prolonged outage due to the unusual spikes of writes.

Everywhere else in Consul we use Jitter liberally to prevent this. We should review the lock library to make sure all waits during the lock acquisition loop have jitter added to prevent this synchronisation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.