hashicorp / hashicorp/vault

raft: Joining node can self-remove permanently during initial log replay

Open
#32,099 2 comments 1 reaction 0 assignees View on GitHub
bug core/ha storage/raft
Dominant language
Go
Stars
36.3k
Forks
4.8k
PR merge metrics
PR metrics pending

Description

**Describe the bug**

The [removed checker](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1479-L1529) introduced in #28726 and #29052 causes node to remove itself permanently during initial log replay on slow or CPU-constrained systems.

**To Reproduce**

Assume a 3-node cluster: `vault-0`, `vault-1` and `vault-2`. The following sequence of events occurs during initial cluster setup:

1. `vault-0` becomes the leader.
2. `vault-1` joins the cluster.
3. `vault-2` joins the cluster.

Log records from the `vault-0` (leader) perspective:

```
$ raft-inspector log LogConfiguration --data-dir data/vault-0
─── Index 1 (raft/raft.db logs/1) ───
Index: 1
Term: 1
Type: LogConfiguration
AppendedAt: 0001-01-01 00:00:00 +0000 UTC
Servers:
- vault-0 (vault-0:8201) voter

─── Index 38 (raft/raft.db logs/38) ───
Index: 38
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:03:46.82551766 +0000 UTC (+0s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) nonvoter

─── Index 289 (raft/raft.db logs/289) ───
Index: 289
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:03:58.502445435 +0000 UTC (+11.677s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) voter

─── Index 290 (raft/raft.db logs/290) ───
Index: 290
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:04:17.093120765 +0000 UTC (+30.268s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) voter
- vault-2 (vault-2:8201) nonvoter
```

4. `vault-2` receives the member list from `vault-0` when joining.
5. `vault-2` calls `raft.BootsrapCluster()` at initial unseal which stores a fabricated entry in the `vault-2` log ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1282), [link](https://github.com/hashicorp/raft/blob/v1.7.3/api.go#L266-L280)).

From `raft.db` on `vault-2`:

```
$ raft-inspector log 1 --data-dir data/vault-2
Index: 1
Term: 1
Type: LogConfiguration
AppendedAt: 0001-01-01 00:00:00 +0000 UTC
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) voter
- vault-2 (vault-2:8201) nonvoter
```

At this point `vault-2` satisfies the first criteria used for self-removal: *`vault-2` has been present in the configuration* ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1510-L1523)).

6. Log entries replicate from `vault-0` to `vault-2`. It processes index 38, where `vault-1` joined.

```
$ raft-inspector log 38 --data-dir data/vault-2
Index: 38
Term: 3
Type: LogConfiguration
AppendedAt: 2026-08-26 08:03:46.82551766 +0000 UTC (+0s)
Servers:
- vault-0 (vault-0:8201) voter
- vault-1 (vault-1:8201) nonvoter
```

At this point, `vault-2`satisfies the second crieria: *it is removed in the configuration* ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L1510-L1523)).

7. The periodic "removed checker" runs and marks the node for removal because both criteria are met.

The CPU running `vault-2` is busy and/or throttled, log replay is slow, and index 290 (where `vault-2` joined) has not been reached yet. As a result, the configuration `vault-2` has still only lists `vault-0` and `vault-1`.

8. Result is permanent shutdown

```
core: shutting down core: error="node has been removed from the HA cluster"
```

`vault-2` has persisted `removed=1` on `raft.db` conf bucket, and it will refuse to work after that ([link](https://github.com/hashicorp/vault/blob/8bce1fbde9b512fecc259ea7ba5bcbcfd6097d40/physical/raft/raft.go#L283-L288))
```
$ raft-inspector status --data-dir data/vault-2 | grep "Removed Self"
Removed Self: 1
```

**Expected behavior**

The cluster should become healthy with all nodes successfully joined.

**Environment:**
* Vault Server Version: 1.19 and later

**Additional context**
I will come up with repro after a bit.

The tool `raft-inspector` is my own, you can find it [here](https://github.com/tsaarni/raft-inspector/).

Contributor guide

Open the contributing guide

Research direction

Start in physical/raft/raft.go at the removed checker around lines 1479–1529 and the BootstrapCluster path around line 1282; compare these with the linked hashicorp/raft API. Use the three-node sequence and raft-inspector logs to examine initial replay before the joining node's configuration entry is reached. Done means a slow or CPU-constrained joining node does not persist Removed Self and the cluster becomes healthy with all nodes joined.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.