hashicorp / hashicorp/consul

Consul server got stuck in "leaving" state on some other servers of the same cluster after maintenance

Open
#13,379 7 comments 8 reactions 0 assignees View on GitHub
theme/federation-usability type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

During maintenance some servers can leave a multi-server cluster (eg shutdown or losing network connectivity using `iptables`). We're experiencing a bug when re-joined servers have `leaving` status on some servers of the cluster but other servers mark them as followers. This seems to be a bug when force-leave operation is not applied.

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a cluster with 9 server nodes
2. Shutdown server/close all ports using iptables, wait
3. Turn on the server/open ports
4. Check `consul operator raft list-peers` on neighbours, some servers can see the server in 'leaving' state while others see this server as follower. Stuck server think that it is a follower
5. Check consul logs on the neighbours, there will be `Initiating push/pull sync with` for wan/lan and everything can seem to be ok

### Consul logs for normally re-joined server and failed server on other servers

re-joined server

```
Jun 01 17:31:24 normal_server consul[39700]: 2022-06-01T17:31:24.719+0300 [ERROR] agent.server: failed to reconcile member: member=“{re_joining_server_info}” error="leadership lost while committing log"
Jun 01 17:31:26 normal_server consul[39700]: 2022-06-01T17:31:26.655+0300 [INFO] agent.server: member joined, marking health alive: member=re_joining_server
Jun 01 17:31:36 normal_server consul[39700]: 2022-06-01T17:31:36.653+0300 [INFO] agent.server.autopilot: Promoting server: id=id address=ip_address:8300 name=re_joining_server
Jun 01 17:31:41 normal_server consul[39700]: 2022-06-01T17:31:41.137+0300 [DEBUG] agent.server.memberlist.wan: memberlist: Initiating push/pull sync with: re_joining_server
Jun 01 17:39:39 normal_server consul[39700]: 2022-06-01T17:39:39.043+0300 [INFO] agent.server: New leader elected: payload=re_joining_server
Jun 01 17:40:43 normal_server consul[39700]: 2022-06-01T17:40:43.517+0300 [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="re_joining_server"
```

During the maintenance the server was in state "left" as if it was force-left by other servers and successfully re-joined the cluster.
The second server was not force-left, but during the maintenance other servers got the message `pinging server failed` and `connection timed out`. Moreover, after some period of time there's message `Rebalanced servers, new active server` on the healthy servers.

### Operating system and Environment details

Ubuntu 20.04, Consul v1.9.5

Contributor guide

Open the contributing guide

Research direction

Reproduce the nine-server scenario by shutting down or blocking a server with iptables, then inspect `consul operator raft list-peers` and the neighbour logs. Compare the servers that report `leaving` with those that report `follower`; this is done when rejoined servers have a consistent status across the cluster and do not remain stuck in `leaving`.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.