hashicorp / hashicorp/vault-helm
Prometheus metrics disappear in HA setup when all Vault pods are sealed
- Dominant language
- Shell
- Stars
- 1.3k
- Forks
- 898
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 1
Description
**Describe the bug**
I'm deploying an HA Vault setup in our Kubernetes cluster with three replicas. While working on monitoring for the seal status of the Vault pods, I noticed that the Prometheus metrics go away when all Vault pods are sealed, which makes it impossible to trigger an alert for this state.
This apparently happens, because the `vault` ServiceMonitor selects the `vault-active` Service, which in turn selects the Vault pod with the `vault-active: "true"` annotation. However, when all Vault pods are sealed, then they all have the `vault-active: "false"` annotation, which means the Service returns 503 when the ServiceMonitor attempts to fetch metrics.
**To Reproduce**
Simply configure Prometheus metrics and then seal all the Vault pods by restarting them
**Expected behavior**
We should be able to get metrics and monitor the seal state via the `vault_core_unsealed` metric even when all Vault pods are sealed.
We achieved this by removing `vault-active: "true"` from the ServiceMonitor `matchLabels` field and adding a new unique label both there and to the `vault` Service object. This ensure the ServiceMonitor uses only the `vault` Service object, which routes to the Vault pods regardless of their active status.
**Environment**
* Kubernetes version: `v1.26.9-eks-a5df82a`
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): EKS
* vault-helm version: `0.25.0`
Chart values:
```yaml
global:
serverTelemetry:
prometheusOperator: true
injector:
enabled: false
server:
ha:
enabled: true
replicas: 3
# Enable HA for integrated storage
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
# Enable unauthenticated metrics access for Prometheus Operator
telemetry {
unauthenticated_metrics_access = "true"
}
}
telemetry {
prometheus_retention_time = "30m"
disable_hostname = true
}
storage "raft" {
path = "/vault/data"
}
# For integrated raft storage and security
# https://developer.hashicorp.com/vault/docs/configuration#disable_mlock
disable_mlock = true
service_registration "kubernetes" {}
serverTelemetry:
serviceMonitor:
enabled: true
dataStorage:
enabled: true
size: 5Gi
storageClass: ebs-gp3
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template "vault.name" . }}
app.kubernetes.io/instance: "{{ .Release.Name }}"
component: server
topologyKey: topology.kubernetes.io/zone
```
Contributor guide
Research direction
Start by inspecting the chart templates for the vault ServiceMonitor and vault Service, using the supplied HA and Prometheus Operator values to reproduce the sealed-pod case. Verify that the ServiceMonitor still reaches Vault pods when all are sealed and that the vault_core_unsealed metric remains available for alerting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, prometheus
- Domain
- devops, infrastructure, observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100