peers.json recovery accepts Raft peer addresses containing a URL scheme, causing failures after leader change
- Dominant language
- Go
- Stars
- 36.3k
- Forks
- 4.8k
- PR merge metrics
- PR metrics pending
Description
**Describe the bug**
When I use a peers.json file to recover my Vault cluster, Vault allows me to put in an erroneous address field including a URL scheme (https), e.g.
```
[
{
"id": "vault-01",
"address": "https://vault-01.example.com:8201",
"non_voter": false
}
]
```
The cluster works fine, the leader comes up and I can join other nodes. The peer list then looks like this:
```
vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-01 https://vault-01.example.com:8201 leader true
vault-02 vault-02.example.com:8201 follower true
vault-03 vault-03.example.com:8201 follower true
```
Yet at some later point in time (weeks later) this will break my cluster after a leader change because the new leader cannot communicate with the node that is registered with the address including https. Error messages from the leader log:
```
[ERROR] storage.raft: failed to appendEntries to: peer="{Voter vault-01 https://vault-01.example.com:8201}" error="dial tcp: address https://vault-01.example.com:8201: too many colons in address"
[ERROR] storage.raft: failed to heartbeat to: peer=https://vault-01.example.com:8201 backoff time=160ms error="dial tcp: address https://vault-01.example.com:8201: too many colons in address"
```
**To Reproduce**
Steps to reproduce the behavior:
1. Stop all Vault nodes, e.g. 01, 02, 03
2. Recover cluster using a peers.json for node 01 according to https://developer.hashicorp.com/vault/tutorials/raft/raft-lost-quorum. Add scheme https to address field.
3. Start 01 as leader and let 02 and 03 join the cluster.
4. Wait a few weeks with Vault running perfectly fine :-)
5. Stop 01; nodes 02 or 03 will become leader.
6. Start 01. It will not be reachable by the leader because of the malformed address.
**Expected behavior**
The `address` field in peers.json should be validated not to include a URL scheme and give an error message if the `address` field has any issues.
**Environment:**
* Vault Server Version (retrieve with `vault status`): 2.0.3
* Vault CLI Version (retrieve with `vault version`): 2.0.3
* Server Operating System/Architecture: Linux/Debian
Vault server configuration file(s):
```hcl
cluster_name = "vault"
api_addr = "https://vault-01.example.com:8200"
cluster_addr = "https://vault-01.example.com:8201"
ui = true
storage "raft" {
path = "/opt/raft"
node_id = "vault-01"
}
disable_mlock = true
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/etc/vault.d/tls/public.cert"
tls_key_file = "/etc/vault.d/tls/private.key"
}
```
**Additional context**
Just as a side note: https://developer.hashicorp.com/vault/tutorials/raft/raft-lost-quorum#view-the-peer-list shows an address field including scheme https.
Contributor guide
Research direction
Start by tracing peers.json recovery and the Raft peer address parsing used when the cluster is restored, then reproduce the malformed https address from the reported recovery steps. The work is done when an address containing a URL scheme is rejected with a clear error and valid peer addresses still recover correctly, with regression coverage for the failure after a leader change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100