hashicorp / hashicorp/consul

Consul cluster fails to recover in outage

Open
#5,546 10 comments 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

The consul cluster has enough nodes to recover, but doesn't.

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a cluster with 3 server nodes
1. Shutdown 2 nodes
1. Create 2 new nodes

1.4.4

I did the aforementioned scenario many times, and sometimes, the remaining survival is elected leader. But mostly, the cluster is broken.

I assume the nodes will eventually return and the cluster will be returned to be healthy, but when the nodes return the last survival, remembers his dead peers, and the election goes in a loop
`consul operator raft list-peers -stale` on the living nodes:
```
consul-bb8f:
Node ID Address State Voter RaftProtocol
consul-hd50 87eabf82-2272-31f3-822c-163f5f8b5cfd xx.xx.0.22:8300 leader true 3
consul-bb8f c80c5e87-e3e1-3c62-a2d5-9ecabe1627c4 xx.xx.0.21:8300 follower false 3
consul-bbl6 be674976-1b0a-3d09-bdc1-087070fb15b4 xx.xx.0.18:8300 follower false 3

consul-hd50
Node ID Address State Voter RaftProtocol
consul-hd50 87eabf82-2272-31f3-822c-163f5f8b5cfd xx.xx.0.22:8300 leader true 3
consul-bb8f c80c5e87-e3e1-3c62-a2d5-9ecabe1627c4 xx.xx.0.21:8300 follower false 3
consul-bbl6 be674976-1b0a-3d09-bdc1-087070fb15b4 xx.0.18:8300 follower false 3

consul-bbl6
Node ID Address State Voter RaftProtocol
consul-28lf a823b90d-c8d7-374d-a137-1978953a9908 xx.xx.0.7:8300 follower true 3
consul-bbl6 be674976-1b0a-3d09-bdc1-087070fb15b4 xx.xx.0.18:8300 follower true 3
```
Constant electiong logs
```
consul-bb8f:
2019/03/25 08:37:03 [ERR] agent: failed to sync remote state: rpc error making call: No cluster leader
2019/03/25 08:37:06 [ERR] raft-net: Failed to flush response: write tcp xx.xx.0.21:8300->xx.xx0.22:55442: write: broken pipe
2019/03/25 08:37:06 [INFO] consul: New leader elected: consul-hd50
2019/03/25 08:37:14 [ERR] http: Request PUT /v1/kv/data, error: rpc error making call: No cluster leader from=127.0.0.1:52808
INFO startup-script: Error! Failed writing data: Unexpected response code: 500 (rpc error making call: No cluster leader)
2019/03/25 08:37:14 [INFO] consul: New leader elected: consul-hd50

consul-hd50
2019/03/25 08:37:14 [INFO] consul: New leader elected: consul-hd50
2019/03/25 08:37:14 [ERR] raft: peer {Nonvoter be674976-1b0a-3d09-bdc1-087070fb15b4 xx.xx.0.18:8300} has newer term, stopping replication
2019/03/25 08:37:14 [INFO] raft: Node at xx.xx.0.22:8300 [Follower] entering Follower state (Leader: "")
2019/03/25 08:37:14 [ERR] consul: failed to wait for barrier: node is not the leader
2019/03/25 08:37:14 [INFO] consul: cluster leadership lost
2019/03/25 08:37:19 [WARN] consul.coordinate: Batch update failed: node is not the leader
2019/03/25 08:37:21 [ERR] agent: failed to sync remote state: No cluster leader
2019/03/25 08:37:23 [WARN] raft: Heartbeat timeout from "" reached, starting election
2019/03/25 08:37:23 [INFO] raft: Node at xx.xx.0.22:8300 [Candidate] entering Candidate state in term 134
2019/03/25 08:37:23 [INFO] raft: Election won. Tally: 1
2019/03/25 08:37:23 [INFO] raft: Node at xx.xx.0.22:8300 [Leader] entering Leader state
2019/03/25 08:37:23 [INFO] raft: Added peer c80c5e87-e3e1-3c62-a2d5-9ecabe1627c4, starting replication
2019/03/25 08:37:23 [INFO] raft: Added peer be674976-1b0a-3d09-bdc1-087070fb15b4, starting replication
2019/03/25 08:37:23 [INFO] consul: cluster leadership acquired
2019/03/25 08:37:23 [INFO] consul: New leader elected: consul-hd50

consul-bbl6
2019/03/25 08:37:55 [INFO] consul: New leader elected: consul-hd50
2019/03/25 08:37:56 [ERR] http: Request GET /v1/kv/stack, error: No cluster leader from=127.0.0.1:49678
2019/03/25 08:37:57 [ERR] agent: failed to sync remote state: No cluster leader
2019/03/25 08:37:59 [WARN] raft: Election timeout reached, restarting election
2019/03/25 08:37:59 [INFO] raft: Node at xx.xx.0.18:8300 [Candidate] entering Candidate state in term 350
2019/03/25 08:37:59 [WARN] raft: Unable to get address for server id a823b90d-c8d7-374d-a137-1978953a9908, using fallback address xx.xx.0.7:8300: Could not find address for server id a823b90d-c8d7-374d-a137-1978953a9908
2019/03/25 08:38:00 [ERR] raft: Failed to make RequestVote RPC to {Voter a823b90d-c8d7-374d-a137-1978953a9908 xx.xx.0.7:8300}: dial tcp ->xx.xx.0.7:8300: i/o timeout
2019/03/25 08:38:01 [INFO] consul: New leader elected: consul-hd50

```

The issue seems to be around the fact that the last survival Request Vote from the dead peer. When I replace the last-survival the cluster is healthy.

I guess this could be recovered manually when such outage occurs, but I'm looking for an automatic solution. How can I detect such a situation?

In some cases the cluster is brought back to be healthy, I'm not sure if It always happens after a period of time, or just randomly. Perhaps after a rebalance event. Trying to reproduce.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.