hashicorp / hashicorp/consul

Cluster leadership does not change when leader's backing disk freezes/becomes unresponsive

Open
#22,064 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

When a consul leader's disk freezes or become unresponsive leadership does not transition to the remaining instances in the cluster. It isn't detected as unhealthy.

**Observed behaviour:** Leadership remains unchanged but all writes to consul hang and eventually fail "agent.server: failed to wait for barrier: error="timed out enqueuing operation""

**Expected behaviour:** Leadership gets removed from the unhealthy instance and another instance in the cluster takes over and handles writes

---

#### Reproduction Steps

1. Start up servers and configure
1. I started three servers in AWS
2. On one server attach an additional volume to be used as Consul's data-dir
1. https://docs.aws.amazon.com/ebs/latest/userguide/ebs-using-volumes.html
2. Install consul on servers
1. https://developer.hashicorp.com/consul/downloads I used the Linux Package Manager instructions and installed v1.20.2
3. Startup consul cluster
1. You'll need the IPs of all 3 servers
2. On the server with the additional volume:
You must fill in `` `` `` and ``
`consul agent -server -data-dir -node frozen_leader -advertise -bind 0.0.0.0 -bootstrap-expect 3 -log-level INFO -retry-join -retry-join -retry-join `
4. On the other two servers run a variation of this command:
You must fill in ` ` `` `` and `` (different for server 2 vs server 3)
`consul agent -server -data-dir /tmp -node -advertise -bind 0.0.0.0 -bootstrap-expect 3 -log-level INFO -retry-join -retry-join -retry-join `
6. Make the server with the extra volume the leader
1. `consul operator raft list-peers`
2. `consul operator raft transfer-leader -id=`
7. Freeze the data dir volume with `fsfreeze`
1. `sudo fsfreeze --freeze `
8. Observer that the leader stays the same, but any write operations, e.g. `consul kv put key value` hang and eventually fail

### Consul info for both Client and Server

Client info

No Clients

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 33e5727a
version = 1.20.2
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.16.0.186:8300
server = true
raft:
applied_index = 2406
commit_index = 2406
fsm_pending = 0
last_contact = 0
last_log_index = 2406
last_log_term = 10
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:1a672c77-928b-0b19-b9e9-c3311459c561 Address:10.16.0.186:8300} {Suffrage:Voter ID:f2eebb9c-8728-c7a5-8832-2e14eed914f8 Address:10.16.0.150:8300} {Suffrage:Voter ID:28f90464-9f81-069f-cb0b-aed769935752 Address:10.16.0.242:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 10
runtime:
arch = amd64
cpu_count = 2
goroutines = 218
max_procs = 2
os = linux
version = go1.22.7
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 10
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 8
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6
members = 3
query_queue = 0
query_time = 1
```

I compared leader vs follower `consul info` output, they're virtually identical so I'll just include the diff of one of them:
```
# diff Leader_consul_info Follower_consul_info
15c15
< leader = true
---
> leader = false
22c22
< last_contact = 0
---
> last_contact = 22.842754ms
35c35
< state = Leader
---
> state = Follower
40c40
< goroutines = 218
---
> goroutines = 144
```

For config see startup command in reproduction steps.

### Operating system and Environment details

Ubuntu 22.04.05 LTS
x86_64

### Log Fragments

I've included the logs seen during an attempted `consul kv put my_data4 123` outside of that there's nothing that looks interesting
These logs occur at the end of the 30 second wait on `kv put`.

```
...
2025-01-09T22:51:28.941Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2025-01-09T22:51:29.727Z [TRACE] agent.server: rpc_server_call: method=KVS.Apply errored=true request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local
2025-01-09T22:51:29.727Z [ERROR] agent.http: Request error: method=PUT url=/v1/kv/my_data4 from=127.0.0.1:15924 error="raft apply failed: timed out enqueuing operation"
2025-01-09T22:51:29.727Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/kv/my_data4 from=127.0.0.1:15924 latency=30.000357704s
2025-01-09T22:51:29.727Z [DEBUG] agent: warning: request content-type is not supported: request-path=/v1/kv/my_data4
2025-01-09T22:51:29.728Z [DEBUG] agent: warning: response content-type header not explicitly set.: request-path=/v1/kv/my_data4
2025-01-09T22:51:30.288Z [TRACE] agent.server.usage_metrics: Starting usage run
2025-01-09T22:51:30.289Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
...
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the three-server setup with the `consul operator raft list-peers`, `consul operator raft transfer-leader`, `fsfreeze`, and `consul kv put` commands described in the issue. Trace the Raft leadership and disk-failure handling from the resulting logs, especially the `raft apply failed: timed out enqueuing operation` path. Done means a frozen leader is removed and another server takes leadership so writes complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.