hashicorp / hashicorp/vault-helm
Example for using raft storage in HA mode not working
- Dominant language
- Shell
- Stars
- 1.3k
- Forks
- 898
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 1
Description
**Describe the bug**
I don't get a functioning Vault cluster when trying to run the example from https://www.vaultproject.io/docs/platform/k8s/helm/examples/ha-with-raft
**To Reproduce**
Steps to reproduce the behavior:
1. Install chart
helm install vault hashicorp/vault --namespace vault --set='server.ha.enabled=true' --set='server.ha.raft.enabled=true'
2. Run vault command
```
kubectl -n vault exec -ti vault-0 -- vault operator init
kubectl -n vault exec -ti vault-0 -- vault operator unseal
kubectl -n vault exec -ti vault-0 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.6.2
Storage Type raft
Cluster Name vault-cluster-fe51402b
Cluster ID f70fe725-0a4b-ea8e-6755-b30743e3e2a6
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address
Raft Committed Index 24
Raft Applied Index 24
kubectl -n vault exec -ti vault-0 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.6.2
Storage Type raft
Cluster Name vault-cluster-fe51402b
Cluster ID f70fe725-0a4b-ea8e-6755-b30743e3e2a6
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Raft Committed Index 30
Raft Applied Index 30
kubectl -n vault exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
Key Value
--- -----
Joined true
kubectl -n vault exec -ti vault-1 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.6.2
Storage Type raft
HA Enabled true
kubectl -n vault exec -ti vault-1 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.6.2
Storage Type raft
HA Enabled true
command terminated with exit code 2
```
3. See error (vault logs, etc.)
Other useful info to include: vault pod logs, `kubectl describe statefulset vault` and `kubectl get statefulset vault -o yaml` output
error in vault-0 pod log:
```2021-04-21T15:32:24.403Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader=
2021-04-21T15:32:24.403Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=19
2021-04-21T15:32:24.425Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
2021-04-21T15:32:28.361Z [INFO] storage.raft: duplicate requestVote for same term: term=19
2021-04-21T15:32:29.796Z [WARN] storage.raft: Election timeout reached, restarting election
2021-04-21T15:32:29.796Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=20
2021-04-21T15:32:29.814Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
2021-04-21T15:32:36.140Z [INFO] storage.raft: duplicate requestVote for same term: term=20
2021-04-21T15:32:36.675Z [WARN] storage.raft: Election timeout reached, restarting election
2021-04-21T15:32:36.675Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=21
2021-04-21T15:32:36.693Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
2021-04-21T15:32:37.021Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2021-04-21T15:32:42.279Z [WARN] storage.raft: Election timeout reached, restarting election
2021-04-21T15:32:42.279Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=22
2021-04-21T15:32:42.321Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
```
error in vault-1 pod log:
```2021-04-21T15:29:45.086Z [INFO] core: security barrier not initialized
2021-04-21T15:29:45.086Z [INFO] core: attempting to join possible raft leader node: leader_addr=http://vault-0.vault-internal:8200
2021-04-21T15:29:46.889Z [INFO] core: security barrier not initialized
2021-04-21T15:29:51.901Z [INFO] core: security barrier not initialized
2021-04-21T15:29:56.893Z [INFO] core: security barrier not initialized
2021-04-21T15:30:01.893Z [INFO] core: security barrier not initialized
2021-04-21T15:30:06.879Z [INFO] core: security barrier not initialized
2021-04-21T15:30:11.907Z [INFO] core: security barrier not initialized
2021-04-21T15:30:16.888Z [INFO] core: security barrier not initialized
2021-04-21T15:30:21.277Z [INFO] core: security barrier not initialized
2021-04-21T15:30:21.277Z [INFO] core: security barrier not initialized
2021-04-21T15:30:21.907Z [INFO] core: security barrier not initialized
2021-04-21T15:30:26.870Z [INFO] core: security barrier not initialized
2021-04-21T15:30:30.932Z [INFO] core: security barrier not initialized
2021-04-21T15:30:30.932Z [INFO] core: security barrier not initialized
2021-04-21T15:30:31.895Z [INFO] core: security barrier not initialized
2021-04-21T15:30:36.877Z [INFO] core: security barrier not initialized
2021-04-21T15:30:39.426Z [INFO] core: security barrier not initialized
2021-04-21T15:30:39.441Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2021-04-21T15:30:39.443Z [ERROR] core.cluster-listener.tcp: error starting listener: error="listen tcp [::]:8201: socket: address family not supported
by protocol"
2021-04-21T15:30:39.457Z [INFO] storage.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:941a20a7-a040-22fa-8f18-6b273fcb8702 Address
:vault-0.vault-internal:8201} {Suffrage:Voter ID:8ad9f007-f4f8-3834-7367-21f433d43e20 Address:vault-1.vault-internal:8201}]"
2021-04-21T15:30:39.458Z [INFO] core: security barrier not initialized
2021-04-21T15:30:39.458Z [INFO] storage.raft: entering follower state: follower="Node at vault-1.vault-internal:8201 [Follower]" leader=
2021-04-21T15:30:41.891Z [INFO] core: security barrier not initialized
2021-04-21T15:30:46.880Z [INFO] core: security barrier not initialized
2021-04-21T15:30:48.804Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader=
2021-04-21T15:30:48.804Z [INFO] storage.raft: entering candidate state: node="Node at vault-1.vault-internal:8201 [Candidate]" term=2
2021-04-21T15:30:48.923Z [INFO] storage.raft: entering follower state: follower="Node at vault-1.vault-internal:8201 [Follower]" leader=
2021-04-21T15:30:51.905Z [INFO] core: security barrier not initialized
2021-04-21T15:30:56.333Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader=
2021-04-21T15:30:56.333Z [INFO] storage.raft: entering candidate state: node="Node at vault-1.vault-internal:8201 [Candidate]" term=4
2021-04-21T15:30:56.353Z [INFO] storage.raft: entering follower state: follower="Node at vault-1.vault-internal:8201 [Follower]" leader=
```
output from `kubectl describe statefulset vault`
```
Name: vault
Namespace: vault
CreationTimestamp: Wed, 21 Apr 2021 17:26:53 +0200
Selector: app.kubernetes.io/instance=vault,app.kubernetes.io/name=vault,component=server
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vault
Annotations:
Replicas: 3 desired | 3 total
Update Strategy: OnDelete
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
helm.sh/chart=vault-0.9.1
Service Account: vault
Containers:
vault:
Image: vault:1.6.2
Ports: 8200/TCP, 8201/TCP, 8202/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-ec
Args:
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl
Readiness: exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
Environment:
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_K8S_POD_NAME: (v1:metadata.name)
VAULT_K8S_NAMESPACE: (v1:metadata.namespace)
VAULT_ADDR: http://127.0.0.1:8200
VAULT_API_ADDR: http://$(POD_IP):8200
SKIP_CHOWN: true
SKIP_SETCAP: true
HOSTNAME: (v1:metadata.name)
VAULT_CLUSTER_ADDR: https://$(HOSTNAME).vault-internal:8201
HOME: /home/vault
Mounts:
/home/vault from home (rw)
/vault/config from config (rw)
/vault/data from data (rw)
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
Volume Claims:
Name: data
StorageClass:
Labels:
Annotations:
Capacity: 10Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 9m4s statefulset-controller create Pod vault-0 in StatefulSet vault successful
Normal SuccessfulCreate 9m3s statefulset-controller create Pod vault-1 in StatefulSet vault successful
Normal SuccessfulCreate 9m2s statefulset-controller create Pod vault-2 in StatefulSet vault successful
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment**
* Kubernetes version: v1.19.6
* deployed with RKE
* vault-helm version: v3.1.2
Contributor guide
Research direction
Start with the HA-with-Raft example linked in the issue and reproduce it using the provided Helm install and kubectl commands. Compare the generated StatefulSet configuration with the vault-0 and vault-1 logs, especially the cluster listener and Raft addresses; done means the documented example forms a functioning HA cluster without the reported connection and listener errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, shell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100