consul lock -monitor-retry failed during election
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
I have 5 Consul servers, quorum is 3. `monitor-retry` did not prevent `consul lock` from failing during election. Running on version 1.4.2.
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Create a cluster with 5 nodes, quorum=3.
2. Run the following on node 1 (non-leader).
```
consul lock -verbose -token=xxxx mylock 'sleep 60'
```
3. Run the following on node 2 (non-leader)
```
consul lock -token=xxxx -monitor-retry 60 mylock 'sleep 1'
```
4. Run the following on leader node.
```
sudo systemctl restart consul
```
### Consul info for both Client and Server
```
agent:
check_monitors = 0
check_ttls = 0
checks = 4
services = 4
build:
prerelease =
revision = c97c712e
version = 1.4.2
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.10.9.164:8300
server = true
raft:
applied_index = 1558164
commit_index = 1558164
fsm_pending = 0
last_contact = 22.8813ms
last_log_index = 1558164
last_log_term = 32
last_snapshot_index = 1549174
last_snapshot_term = 20
latest_configuration = ......
latest_configuration_index = 1558146
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
term = 32
runtime:
arch = amd64
cpu_count = 2
goroutines = 87
max_procs = 10
os = linux
version = go1.11.4
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 16
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 40
members = 5
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 127
members = 5
query_queue = 0
query_time = 1
```
### Operating system and Environment details
CentOS Linux release 7.6.1810 (Core)
### Log Fragments
On node 2 (where `monitor-retry` is used). This failure occurs pretty much within the default timeout (which is 3 seconds according to the doc).
```
[myuser@myhost ~]$ consul lock -token=xxxx -monitor-retry 60 mylock 'sleep 1'
Lock acquisition failed: failed to read lock: Unexpected response code: 500
```
Contributor guide
Assessment
This issue has not been assessed yet.