Server restart leads to client leader election issues
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
### `consul version` for both Client and Server
Client: `Go API Client, HEAD`
Server: `0.7.0`
### Operating system and Environment details
`Ubuntu Linux 14.04`
### Description of the Issue (and unexpected/desired result)
We are using client side leader election (https://github.com/hashicorp/consul/blob/d5b7530ec593f1ec2a8f8a7c145bcadafa88b572/api/lock.go#L132)
When leadership is acquired, two goroutines are created:
https://github.com/hashicorp/consul/blob/d5b7530ec593f1ec2a8f8a7c145bcadafa88b572/api/lock.go#L151
https://github.com/hashicorp/consul/blob/d5b7530ec593f1ec2a8f8a7c145bcadafa88b572/api/lock.go#L239
When the current consul server leader is restarted, the monitor goroutine fails I assume because it is querying in consistent mode. This signals to the client that leadership was revoked. I would assume in this case to try and acquire leadership again since this is a blocking process. However, the renew goroutine started earlier continues to renew the old session which means that the lock appears to be held.
Our current workaround is to call Unlock() when our leadership is revoked but this wasn't obvious.
Is this behaviour expected? If so, it should be documented.
If not, I propose that the sessionRenew channel on the lock should be closed when leadership is revoked.
### Reproduction steps
Launch a 3 node cluster. (We used docker)
Client code should attempt to acquire a leadership and then wait for it to be revoked.
Restart the consul server (docker restart work well)
Once revoked, the client code should loop around and try to acquire leadership again.
Instead, it waits forever and lock is not required.
If the consul servers are run with DEBUG logging you'll see that the old and new lock sessions are both being renewed.
Contributor guide
Assessment
This issue has not been assessed yet.