hashicorp / hashicorp/vault-helm

Example for using raft storage in HA mode not working

Open
#503 2 comments 4 reactions 0 assignees View on GitHub
bug rancher
Dominant language
Shell
Stars
1.3k
Forks
898
Avg merge
3d 1h
Merged PRs (30d)
1

Description

**Describe the bug**
I don't get a functioning Vault cluster when trying to run the example from https://www.vaultproject.io/docs/platform/k8s/helm/examples/ha-with-raft

**To Reproduce**
Steps to reproduce the behavior:
1. Install chart
helm install vault hashicorp/vault --namespace vault --set='server.ha.enabled=true' --set='server.ha.raft.enabled=true'
2. Run vault command
```
kubectl -n vault exec -ti vault-0 -- vault operator init
kubectl -n vault exec -ti vault-0 -- vault operator unseal
kubectl -n vault exec -ti vault-0 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.6.2
Storage Type raft
Cluster Name vault-cluster-fe51402b
Cluster ID f70fe725-0a4b-ea8e-6755-b30743e3e2a6
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address
Raft Committed Index 24
Raft Applied Index 24

kubectl -n vault exec -ti vault-0 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.6.2
Storage Type raft
Cluster Name vault-cluster-fe51402b
Cluster ID f70fe725-0a4b-ea8e-6755-b30743e3e2a6
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Raft Committed Index 30
Raft Applied Index 30

kubectl -n vault exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
Key Value
--- -----
Joined true

kubectl -n vault exec -ti vault-1 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.6.2
Storage Type raft
HA Enabled true

kubectl -n vault exec -ti vault-1 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.6.2
Storage Type raft
HA Enabled true
command terminated with exit code 2

```

3. See error (vault logs, etc.)

Other useful info to include: vault pod logs, `kubectl describe statefulset vault` and `kubectl get statefulset vault -o yaml` output
error in vault-0 pod log:
```2021-04-21T15:32:24.403Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader=
2021-04-21T15:32:24.403Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=19
2021-04-21T15:32:24.425Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
2021-04-21T15:32:28.361Z [INFO] storage.raft: duplicate requestVote for same term: term=19
2021-04-21T15:32:29.796Z [WARN] storage.raft: Election timeout reached, restarting election
2021-04-21T15:32:29.796Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=20
2021-04-21T15:32:29.814Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
2021-04-21T15:32:36.140Z [INFO] storage.raft: duplicate requestVote for same term: term=20
2021-04-21T15:32:36.675Z [WARN] storage.raft: Election timeout reached, restarting election
2021-04-21T15:32:36.675Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=21
2021-04-21T15:32:36.693Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
2021-04-21T15:32:37.021Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2021-04-21T15:32:42.279Z [WARN] storage.raft: Election timeout reached, restarting election
2021-04-21T15:32:42.279Z [INFO] storage.raft: entering candidate state: node="Node at vault-0.vault-internal:8201 [Candidate]" term=22
2021-04-21T15:32:42.321Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 8ad9f007-f4f8-3834-7367-21f433d43e20 vault-1.vault-internal:8201}" error="dial tcp 10.42.1.158:8201: connect: connection refused"
```

error in vault-1 pod log:

```2021-04-21T15:29:45.086Z [INFO] core: security barrier not initialized
2021-04-21T15:29:45.086Z [INFO] core: attempting to join possible raft leader node: leader_addr=http://vault-0.vault-internal:8200
2021-04-21T15:29:46.889Z [INFO] core: security barrier not initialized
2021-04-21T15:29:51.901Z [INFO] core: security barrier not initialized
2021-04-21T15:29:56.893Z [INFO] core: security barrier not initialized
2021-04-21T15:30:01.893Z [INFO] core: security barrier not initialized
2021-04-21T15:30:06.879Z [INFO] core: security barrier not initialized
2021-04-21T15:30:11.907Z [INFO] core: security barrier not initialized
2021-04-21T15:30:16.888Z [INFO] core: security barrier not initialized
2021-04-21T15:30:21.277Z [INFO] core: security barrier not initialized
2021-04-21T15:30:21.277Z [INFO] core: security barrier not initialized
2021-04-21T15:30:21.907Z [INFO] core: security barrier not initialized
2021-04-21T15:30:26.870Z [INFO] core: security barrier not initialized
2021-04-21T15:30:30.932Z [INFO] core: security barrier not initialized
2021-04-21T15:30:30.932Z [INFO] core: security barrier not initialized
2021-04-21T15:30:31.895Z [INFO] core: security barrier not initialized
2021-04-21T15:30:36.877Z [INFO] core: security barrier not initialized
2021-04-21T15:30:39.426Z [INFO] core: security barrier not initialized
2021-04-21T15:30:39.441Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2021-04-21T15:30:39.443Z [ERROR] core.cluster-listener.tcp: error starting listener: error="listen tcp [::]:8201: socket: address family not supported
by protocol"
2021-04-21T15:30:39.457Z [INFO] storage.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:941a20a7-a040-22fa-8f18-6b273fcb8702 Address
:vault-0.vault-internal:8201} {Suffrage:Voter ID:8ad9f007-f4f8-3834-7367-21f433d43e20 Address:vault-1.vault-internal:8201}]"
2021-04-21T15:30:39.458Z [INFO] core: security barrier not initialized
2021-04-21T15:30:39.458Z [INFO] storage.raft: entering follower state: follower="Node at vault-1.vault-internal:8201 [Follower]" leader=
2021-04-21T15:30:41.891Z [INFO] core: security barrier not initialized
2021-04-21T15:30:46.880Z [INFO] core: security barrier not initialized
2021-04-21T15:30:48.804Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader=
2021-04-21T15:30:48.804Z [INFO] storage.raft: entering candidate state: node="Node at vault-1.vault-internal:8201 [Candidate]" term=2
2021-04-21T15:30:48.923Z [INFO] storage.raft: entering follower state: follower="Node at vault-1.vault-internal:8201 [Follower]" leader=
2021-04-21T15:30:51.905Z [INFO] core: security barrier not initialized
2021-04-21T15:30:56.333Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader=
2021-04-21T15:30:56.333Z [INFO] storage.raft: entering candidate state: node="Node at vault-1.vault-internal:8201 [Candidate]" term=4
2021-04-21T15:30:56.353Z [INFO] storage.raft: entering follower state: follower="Node at vault-1.vault-internal:8201 [Follower]" leader=
```

output from `kubectl describe statefulset vault`

```
Name: vault
Namespace: vault
CreationTimestamp: Wed, 21 Apr 2021 17:26:53 +0200
Selector: app.kubernetes.io/instance=vault,app.kubernetes.io/name=vault,component=server
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vault
Annotations:
Replicas: 3 desired | 3 total
Update Strategy: OnDelete
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
helm.sh/chart=vault-0.9.1
Service Account: vault
Containers:
vault:
Image: vault:1.6.2
Ports: 8200/TCP, 8201/TCP, 8202/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-ec
Args:
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

Readiness: exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
Environment:
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_K8S_POD_NAME: (v1:metadata.name)
VAULT_K8S_NAMESPACE: (v1:metadata.namespace)
VAULT_ADDR: http://127.0.0.1:8200
VAULT_API_ADDR: http://$(POD_IP):8200
SKIP_CHOWN: true
SKIP_SETCAP: true
HOSTNAME: (v1:metadata.name)
VAULT_CLUSTER_ADDR: https://$(HOSTNAME).vault-internal:8201
HOME: /home/vault
Mounts:
/home/vault from home (rw)
/vault/config from config (rw)
/vault/data from data (rw)
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
Volume Claims:
Name: data
StorageClass:
Labels:
Annotations:
Capacity: 10Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 9m4s statefulset-controller create Pod vault-0 in StatefulSet vault successful
Normal SuccessfulCreate 9m3s statefulset-controller create Pod vault-1 in StatefulSet vault successful
Normal SuccessfulCreate 9m2s statefulset-controller create Pod vault-2 in StatefulSet vault successful
```

**Expected behavior**
A clear and concise description of what you expected to happen.

**Environment**
* Kubernetes version: v1.19.6
* deployed with RKE

* vault-helm version: v3.1.2

Contributor guide

Open the contributing guide

Research direction

Start with the HA-with-Raft example linked in the issue and reproduce it using the provided Helm install and kubectl commands. Compare the generated StatefulSet configuration with the vault-0 and vault-1 logs, especially the cluster listener and Raft addresses; done means the documented example forms a functioning HA cluster without the reported connection and listener errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.