raft: Joining node can self-remove permanently during initial log replay
- Dominant language
- Go
- Stars
- 36.3k
- Forks
- 4.8k
- PR merge metrics
- PR metrics pending
Description
**Describe the bug**
The [removed checker](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1479-L1529) introduced in #28726 and #29052 causes node to remove itself permanently during initial log replay on slow or CPU-constrained systems.
**To Reproduce**
Assume a 3-node cluster: `vault-0`, `vault-1` and `vault-2`. The following sequence of events occurs during initial cluster setup:
1. `vault-0` becomes the leader.
2. `vault-1` joins the cluster.
3. `vault-2` joins the cluster.
Log records from the `vault-0` (leader) perspective:
```
$ raft-inspector log LogConfiguration --data-dir data/vault-0
─── Index 1 (raft/raft.db logs/1) ───
Index: 1
Term: 1
Type: LogConfiguration
AppendedAt: 0001-01-01 00:00:00 +0000 UTC
Servers:
- vault-0 (vault-0:8201) voter
─── Index 38 (raft/raft.db logs/38) ───
Index: 38
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:03:46.82551766 +0000 UTC (+0s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) nonvoter
─── Index 289 (raft/raft.db logs/289) ───
Index: 289
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:03:58.502445435 +0000 UTC (+11.677s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) voter
─── Index 290 (raft/raft.db logs/290) ───
Index: 290
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:04:17.093120765 +0000 UTC (+30.268s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) voter
- vault-2 (vault-2:8201) nonvoter
```
4. `vault-2` receives the member list from `vault-0` when joining.
5. `vault-2` calls `raft.BootsrapCluster()` at initial unseal which stores a fabricated entry in the `vault-2` log ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1282), [link](https://github.com/hashicorp/raft/blob/v1.7.3/api.go#L266-L280)).
From `raft.db` on `vault-2`:
```
$ raft-inspector log 1 --data-dir data/vault-2
Index: 1
Term: 1
Type: LogConfiguration
AppendedAt: 0001-01-01 00:00:00 +0000 UTC
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) voter
- vault-2 (vault-2:8201) nonvoter
```
At this point `vault-2` satisfies the first criteria used for self-removal: *`vault-2` has been present in the configuration* ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1510-L1523)).
6. Log entries replicate from `vault-0` to `vault-2`. It processes index 38, where `vault-1` joined.
```
$ raft-inspector log 38 --data-dir data/vault-2
Index: 38
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:03:46.82551766 +0000 UTC (+0s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) nonvoter
```
At this point, `vault-2`satisfies the second crieria: *it is removed in the configuration* ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1510-L1523)).
7. The periodic "removed checker" runs and marks the node for removal because both criteria are met.
The CPU running `vault-2` is busy and/or throttled, log replay is slow, and index 290 (where `vault-2` joined) has not been reached yet. As a result, the configuration `vault-2` has still only lists `vault-0` and `vault-1`.
8. Result is permanent shutdown
```
core: shutting down core: error="node has been removed from the HA cluster"
```
`vault-2` has persisted `removed=1` on `raft.db` conf bucket, and it will refuse to work after that ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L283-L288))
```
$ raft-inspector status --data-dir data/vault-2 | grep "Removed Self"
Removed Self: 1
```
**Expected behavior**
The cluster should become healthy with all nodes successfully joined.
**Environment:**
* Vault Server Version: 1.19 and later
**Additional context**
I will come up with repro after a bit.
The tool `raft-inspector` is my own, you can find it [here](https://github.com/tsaarni/raft-inspector/).
Contributor guide
Research direction
Start in physical/raft/raft.go at the removed checker around lines 1479–1529 and the BootstrapCluster path around line 1282; compare these with the linked hashicorp/raft API. Use the three-node sequence and raft-inspector logs to examine initial replay before the joining node's configuration entry is reached. Done means a slow or CPU-constrained joining node does not persist Removed Self and the cluster becomes healthy with all nodes joined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100