hashicorp / hashicorp/vault-helm

Prometheus metrics disappear in HA setup when all Vault pods are sealed

Open
#990 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Shell
Stars
1.3k
Forks
898
Avg merge
3d 1h
Merged PRs (30d)
1

Description

**Describe the bug**
I'm deploying an HA Vault setup in our Kubernetes cluster with three replicas. While working on monitoring for the seal status of the Vault pods, I noticed that the Prometheus metrics go away when all Vault pods are sealed, which makes it impossible to trigger an alert for this state.

This apparently happens, because the `vault` ServiceMonitor selects the `vault-active` Service, which in turn selects the Vault pod with the `vault-active: "true"` annotation. However, when all Vault pods are sealed, then they all have the `vault-active: "false"` annotation, which means the Service returns 503 when the ServiceMonitor attempts to fetch metrics.

**To Reproduce**
Simply configure Prometheus metrics and then seal all the Vault pods by restarting them

**Expected behavior**
We should be able to get metrics and monitor the seal state via the `vault_core_unsealed` metric even when all Vault pods are sealed.

We achieved this by removing `vault-active: "true"` from the ServiceMonitor `matchLabels` field and adding a new unique label both there and to the `vault` Service object. This ensure the ServiceMonitor uses only the `vault` Service object, which routes to the Vault pods regardless of their active status.

**Environment**
* Kubernetes version: `v1.26.9-eks-a5df82a`
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): EKS
* vault-helm version: `0.25.0`

Chart values:

```yaml
global:
serverTelemetry:
prometheusOperator: true
injector:
enabled: false
server:
ha:
enabled: true
replicas: 3
# Enable HA for integrated storage
raft:
enabled: true
setNodeId: true
config: |
ui = true

listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"

# Enable unauthenticated metrics access for Prometheus Operator
telemetry {
unauthenticated_metrics_access = "true"
}
}

telemetry {
prometheus_retention_time = "30m"
disable_hostname = true
}

storage "raft" {
path = "/vault/data"
}

# For integrated raft storage and security
# https://developer.hashicorp.com/vault/docs/configuration#disable_mlock
disable_mlock = true

service_registration "kubernetes" {}
serverTelemetry:
serviceMonitor:
enabled: true
dataStorage:
enabled: true
size: 5Gi
storageClass: ebs-gp3
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template "vault.name" . }}
app.kubernetes.io/instance: "{{ .Release.Name }}"
component: server
topologyKey: topology.kubernetes.io/zone

```

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the chart templates for the vault ServiceMonitor and vault Service, using the supplied HA and Prometheus Operator values to reproduce the sealed-pod case. Verify that the ServiceMonitor still reaches Vault pods when all are sealed and that the vault_core_unsealed metric remains available for alerting.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, prometheus
Domain
devops, infrastructure, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.