hashicorp / hashicorp/vault-helm
K8s Vault docker can't start pod log says : Error initializing listener of type tcp: error loading TLS cert: open : no such file or directory
- Dominant language
- Shell
- Stars
- 1.3k
- Forks
- 897
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 1
Description
**Describe the bug**
I created k8s secret and followed the manual on how to create [TLS certificates](https://github.com/hashicorp/vault/blob/550f08648552acea274245c5b864052889cde236/website/source/docs/platform/k8s/helm/examples.html.md#helm-configuration)
first, i create the secret in k8s then
i deploy vault with ha configuration and raft storage
bug pods are with error that looks like this :
```
Error initializing listener of type tcp: error loading TLS cert: open : no such file or directory
```
Steps to reproduce the behavior:
1. Install chart
2. Run vault command
3. See error (vault logs, etc.)
```
kubectl get secret vault-server-tls -n vault-foo -o yaml
apiVersion: v1
data:
vault.ca: LS0t.....GSUNBVEUtLS0tLQo=
vault.crt: LS0tLS1CRUdJ....LS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
vault.key: LS0tLS1CR....S0tDQo=
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","data":{"vault.ca":"LS0tLS1CR...FWS0tLS0tDQo="},"kind":"Secret","metadata":{"annotations":{},"creationTimestamp":"2021-01-21T09:34:31Z","managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:data":{".":{},"f:vault.ca":{},"f:vault.crt":{},"f:vault.key":{}},"f:type":{}},"manager":"kubectl.exe","operation":"Update","time":"2021-01-21T09:34:31Z"}],"name":"vault-server-tls","namespace":"vault-foo","selfLink":"/api/v1/namespaces/vault-foo/secrets/vault-server-tls","uid":"845b856e-d934-46dd-b094-ca75084542cd"},"type":"Opaque"}
creationTimestamp: "2021-01-21T09:34:31Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:vault.ca: {}
f:vault.crt: {}
f:vault.key: {}
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:type: {}
manager: kubectl.exe
operation: Update
time: "2021-01-21T09:39:10Z"
name: vault-server-tls
namespace: vault-foo
resourceVersion: "62302347"
selfLink: /api/v1/namespaces/vault-foo/secrets/vault-server-tls
uid: 845b856e-d934-46dd-b094-ca75084542cd
type: Opaque
```
```
$ kubectl describe statefulset vault -n vault-foo
Name: vault
Namespace: vault-foo
CreationTimestamp: Thu, 21 Jan 2021 16:42:16 +0200
Selector: app.kubernetes.io/instance=vault,app.kubernetes.io/name=vault,component=server
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vault
Annotations: meta.helm.sh/release-name: vault
meta.helm.sh/release-namespace: vault-foo
Replicas: 3 desired | 3 total
Update Strategy: OnDelete
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
helm.sh/chart=vault-0.9.0
Service Account: vault
Containers:
vault:
Image: hashicorp/vault-enterprise:1.5.0_ent
Ports: 8200/TCP, 8201/TCP, 8202/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-ec
Args:
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl
Limits:
cpu: 2
memory: 16Gi
Requests:
cpu: 2
memory: 8Gi
Liveness: http-get https://:8200/v1/sys/health%3Fstandbyok=true delay=60s timeout=3s period=5s #success=1 #failure=2
Readiness: http-get https://:8200/v1/sys/health%3Fstandbyok=true&sealedcode=204&uninitcode=204 delay=5s timeout=3s period=5s #success=1 #failure=2
Environment:
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_K8S_POD_NAME: (v1:metadata.name)
VAULT_K8S_NAMESPACE: (v1:metadata.namespace)
VAULT_ADDR: https://127.0.0.1:8200
VAULT_API_ADDR: https://$(POD_IP):8200
SKIP_CHOWN: true
SKIP_SETCAP: true
HOSTNAME: (v1:metadata.name)
VAULT_CLUSTER_ADDR: https://$(HOSTNAME).vault-internal:8201
VAULT_RAFT_NODE_ID: (v1:metadata.name)
HOME: /home/vault
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.crt
Mounts:
/home/vault from home (rw)
/vault/audit from audit (rw)
/vault/config from config (rw)
/vault/data from data (rw)
/vault/userconfig/vault-server-tls from userconfig-vault-server-tls (ro)
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
userconfig-vault-server-tls:
Type: Secret (a volume populated by a Secret)
SecretName: vault-server-tls
Optional: false
home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
Volume Claims:
Name: data
StorageClass:
Labels:
Annotations:
Capacity: 10Gi
Access Modes: [ReadWriteOnce]
Name: audit
StorageClass:
Labels:
Annotations:
Capacity: 10Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 40m statefulset-controller create Pod vault-0 in StatefulSet vault successful
Normal SuccessfulCreate 40m statefulset-controller create Pod vault-1 in StatefulSet vault successful
Normal SuccessfulCreate 40m statefulset-controller create Pod vault-2 in StatefulSet vault successful
```
**Environment**
* Kubernetes version: Aws EKS
* vault-helm version: v3.2.2"
Chart values:
```yaml
$ helm get values vault -n vault-foo
USER-SUPPLIED VALUES:
global:
enabled: true
tlsDisable: false
injector:
enabled: true
image:
repository: hashicorp/vault-k8s
tag: latest
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 250m
memory: 256Mi
server:
auditStorage:
enabled: true
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.crt
extraVolumes:
- name: vault-server-tls
type: secret
ha:
enabled: true
raft:
config: |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
#tls_disable = 1
}
storage "raft" {
path = "/vault/data"
retry_join {
leader_api_addr = "http://vault-0.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
}
retry_join {
leader_api_addr = "http://vault-1.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
leader_client_cert_file = "/vault/userconfig/vault-server-tlsr/vault.crt"
leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
}
retry_join {
leader_api_addr = "http://vault-2.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
}
}
service_registration "kubernetes" {}
enabled: true
setNodeId: true
replicas: 3
image:
repository: hashicorp/vault-enterprise
tag: 1.5.0_ent
livenessProbe:
enabled: true
initialDelaySeconds: 60
path: /v1/sys/health?standbyok=true
readinessProbe:
enabled: true
path: /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
resources:
limits:
cpu: 2000m
memory: 16Gi
requests:
cpu: 2000m
memory: 8Gi
standalone:
enabled: false
ui:
enabled: true
externalPort: 8200
serviceNodePort: null
serviceType: LoadBalancer
```
The dockers are not started as I'm getting :
```
$ kubectl exec -it vault-0 -n vault-foo – /bin/sh
Unable to use a TTY - input is not a terminal or the right kind of file
error: unable to upgrade connection: container not found ("vault")
$ kubectl exec -i -t vault-0 -n vault-foo – ls /vault/userconfig/vault-server-tls
Unable to use a TTY - input is not a terminal or the right kind of file
error: unable to upgrade connection: container not found ("vault")
```
but the pods are installed
```
$ kubectl get pods -n vault-foo
NAME READY STATUS RESTARTS AGE
vault-0 0/1 CrashLoopBackOff 5 3m36s
vault-1 0/1 CrashLoopBackOff 5 3m36s
vault-2 0/1 CrashLoopBackOff 5 3m36s
vault-agent-injector-d54bdc675-79ll7 1/1 Running 0 3m36s
```
what I'm missing here?
Contributor guide
Research direction
Start with the linked Helm TLS configuration example, the supplied raft config, and the rendered StatefulSet from `kubectl describe statefulset`. Compare the secret-mounted paths and listener certificate settings with the rendered configuration, then verify the result with `kubectl logs` and a pod restart. Done means all Vault pods start without the TLS listener error and reach Ready.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, helm, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100