Consul server got stuck in "leaving" state on some other servers of the same cluster after maintenance
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
During maintenance some servers can leave a multi-server cluster (eg shutdown or losing network connectivity using `iptables`). We're experiencing a bug when re-joined servers have `leaving` status on some servers of the cluster but other servers mark them as followers. This seems to be a bug when force-leave operation is not applied.
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Create a cluster with 9 server nodes
2. Shutdown server/close all ports using iptables, wait
3. Turn on the server/open ports
4. Check `consul operator raft list-peers` on neighbours, some servers can see the server in 'leaving' state while others see this server as follower. Stuck server think that it is a follower
5. Check consul logs on the neighbours, there will be `Initiating push/pull sync with` for wan/lan and everything can seem to be ok
### Consul logs for normally re-joined server and failed server on other servers
re-joined server
```
Jun 01 17:31:24 normal_server consul[39700]: 2022-06-01T17:31:24.719+0300 [ERROR] agent.server: failed to reconcile member: member=“{re_joining_server_info}” error="leadership lost while committing log"
Jun 01 17:31:26 normal_server consul[39700]: 2022-06-01T17:31:26.655+0300 [INFO] agent.server: member joined, marking health alive: member=re_joining_server
Jun 01 17:31:36 normal_server consul[39700]: 2022-06-01T17:31:36.653+0300 [INFO] agent.server.autopilot: Promoting server: id=id address=ip_address:8300 name=re_joining_server
Jun 01 17:31:41 normal_server consul[39700]: 2022-06-01T17:31:41.137+0300 [DEBUG] agent.server.memberlist.wan: memberlist: Initiating push/pull sync with: re_joining_server
Jun 01 17:39:39 normal_server consul[39700]: 2022-06-01T17:39:39.043+0300 [INFO] agent.server: New leader elected: payload=re_joining_server
Jun 01 17:40:43 normal_server consul[39700]: 2022-06-01T17:40:43.517+0300 [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="re_joining_server"
```
During the maintenance the server was in state "left" as if it was force-left by other servers and successfully re-joined the cluster.
The second server was not force-left, but during the maintenance other servers got the message `pinging server failed` and `connection timed out`. Moreover, after some period of time there's message `Rebalanced servers, new active server` on the healthy servers.
### Operating system and Environment details
Ubuntu 20.04, Consul v1.9.5
Contributor guide
Research direction
Reproduce the nine-server scenario by shutting down or blocking a server with iptables, then inspect `consul operator raft list-peers` and the neighbour logs. Compare the servers that report `leaving` with those that report `follower`; this is done when rejoined servers have a consistent status across the cluster and do not remain stuck in `leaving`.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100