hashicorp / hashicorp/vault-helm

Pods from same replicaset have different CA Bundles

Open
#580 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Shell
Stars
1.3k
Forks
898
Avg merge
3d 1h
Merged PRs (30d)
1

Description

**Describe the bug**
When running the vault injector chart with >1 replicas & auto-tls, we occasionally find that some of our vault injector replicas are using different CAs than the other replicas within the same deployments. This leads to them fighting over the value of the CABundle in Webhook spec, which is what the Controller Manager uses when verifying a replica's identity. The injector is then only partially functional, with the injector pods that have the incorrect CA bundles not working. The pods and their containers continue in the "running" phase, even with the error.

Here are error logs from the controller manager when it tries to connect to an injector pod with a different CA from the one set in the webhook spec:

`
E0219 15:31:38.889156 1 replica_set.go:536] sync "/nginx-ingress-nginx-ingress-controller-59f4fb5c74" failed with Internal error occurred: failed calling webhook "vault.hashicorp.com": Post https://vault-agent-injector-svc.vault.svc:443/mutate?timeout=10s: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "Agent Inject CA")
`

This seems to happen most when the vault injector pods are created at different times. Deleting all pods in the replicaset temporarily fixes the issue.

**To Reproduce**
Steps to reproduce the behavior:
1. Install chart with multiple replicas
2. Delete one or more pods at different times or wait until some pods go down
3. See error, occasionally

**Expected behavior**
We expect all replicas of the vault-injector deployment to hold the same CA bundle at all times.

**Environment**
* Kubernetes version: 1.18.2
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): None
* Other configuration options or runtime services (istio, etc.): None
* vault-helm version: 0.9.0
* vault injector version 0.7.0

Chart values:

```yaml
global:
tlsDisable: false
injector:
enabled: true
replicas: 3
leaderElector:
image:
repository: /gcr.io/google_containers/leader-elector
metrics:
enabled: true
externalVaultAddr:
image:
repository:/registry-1.docker.io/hashicorp/vault-k8s
agentImage:
repository: /registry-1.docker.io/hashicorp/vault
authPath: "auth/kubernetes/"
logFormat: "json"
namespaceSelector:
matchExpressions:
- key: vault-injector-webhook
operator: NotIn
values: ["disabled"]
failurePolicy: Fail
tolerations: |
- effect: NoSchedule
key: node-role.kubernetes.io/master
nodeSelector: |
beta.kubernetes.io/os: linux
node-role.kubernetes.io/master: ""
```

**Additional context**
It looks like some sort of similar issue was fixed in https://github.com/hashicorp/vault-k8s/issues/141, but we are using the versions of the injector an the helm chart where this was fixed according to that page.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the provided chart values, three replicas, and pods deleted at different times. Review the injector deployment and webhook behavior alongside the controller-manager x509 error, then verify that all replicas retain the same CA bundle and that staggered restarts no longer cause webhook failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.