hashicorp / hashicorp/vault

peers.json recovery accepts Raft peer addresses containing a URL scheme, causing failures after leader change

Open
#32,100 1 comment 1 reaction 0 assignees View on GitHub
bug storage/raft
Dominant language
Go
Stars
36.3k
Forks
4.8k
PR merge metrics
PR metrics pending

Description

**Describe the bug**
When I use a peers.json file to recover my Vault cluster, Vault allows me to put in an erroneous address field including a URL scheme (https), e.g.

```
[
{
"id": "vault-01",
"address": "https://vault-01.example.com:8201",
"non_voter": false
}
]
```

The cluster works fine, the leader comes up and I can join other nodes. The peer list then looks like this:

```
vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-01 https://vault-01.example.com:8201 leader true
vault-02 vault-02.example.com:8201 follower true
vault-03 vault-03.example.com:8201 follower true
```

Yet at some later point in time (weeks later) this will break my cluster after a leader change because the new leader cannot communicate with the node that is registered with the address including https. Error messages from the leader log:

```
[ERROR] storage.raft: failed to appendEntries to: peer="{Voter vault-01 https://vault-01.example.com:8201}" error="dial tcp: address https://vault-01.example.com:8201: too many colons in address"
[ERROR] storage.raft: failed to heartbeat to: peer=https://vault-01.example.com:8201 backoff time=160ms error="dial tcp: address https://vault-01.example.com:8201: too many colons in address"

```
**To Reproduce**
Steps to reproduce the behavior:
1. Stop all Vault nodes, e.g. 01, 02, 03
2. Recover cluster using a peers.json for node 01 according to https://developer.hashicorp.com/vault/tutorials/raft/raft-lost-quorum. Add scheme https to address field.
3. Start 01 as leader and let 02 and 03 join the cluster.
4. Wait a few weeks with Vault running perfectly fine :-)
5. Stop 01; nodes 02 or 03 will become leader.
6. Start 01. It will not be reachable by the leader because of the malformed address.

**Expected behavior**
The `address` field in peers.json should be validated not to include a URL scheme and give an error message if the `address` field has any issues.

**Environment:**
* Vault Server Version (retrieve with `vault status`): 2.0.3
* Vault CLI Version (retrieve with `vault version`): 2.0.3
* Server Operating System/Architecture: Linux/Debian

Vault server configuration file(s):

```hcl
cluster_name = "vault"

api_addr = "https://vault-01.example.com:8200"
cluster_addr = "https://vault-01.example.com:8201"

ui = true

storage "raft" {
path = "/opt/raft"
node_id = "vault-01"
}

disable_mlock = true
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/etc/vault.d/tls/public.cert"
tls_key_file = "/etc/vault.d/tls/private.key"
}

```

**Additional context**
Just as a side note: https://developer.hashicorp.com/vault/tutorials/raft/raft-lost-quorum#view-the-peer-list shows an address field including scheme https.

Contributor guide

Open the contributing guide

Research direction

Start by tracing peers.json recovery and the Raft peer address parsing used when the cluster is restored, then reproduce the malformed https address from the reported recovery steps. The work is done when an address containing a URL scheme is rejected with a clear error and valid peer addresses still recover correctly, with regression coverage for the failure after a leader change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.