hashicorp / hashicorp/vault-secrets-operator
VSO controller_resource_status metrics not updating
- Dominant language
- Go
- Stars
- 600
- Forks
- 146
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 6
Description
**Describe the bug**
The `controller_resource_status` metrics, which contain metrics on the auth and connections status from the VSO to a Vault instance do not get updated.
Metrics for `controller="vaultconnection"` and `controller="vaultauth"` experience this issue.
From the observed behavior, the metric only gets updated when going from a failure state to a working state, the other way around doesn't happen.
**To Reproduce**
Steps to reproduce the behavior:
1. Deploy the Vault Secrets Operator to the Kubernetes cluster with proper connection configurations in place, like network policies;
2. Wait for the VSO to start, run the metric related checks and start exposing the metrics;
3. Validate that the `controller_resource_status` metrics show a `1` value, meaning that the VSO was able to connect to Vault;
4. Drop the network policies (or network connectivity) to Vault;
5. Watch the metrics, and see that they are not being update to a `0` value, meaning that the connection is failing.
Here is a screenshot of a dashboard to visualize the behavior


Application deployment:
There is no application involved in this case, since this is a VSO issue.
**Expected behavior**
When the connectivity to Vault becomes unavailable, the metrics should be updated to show the actual status.
**Environment**
* Kubernetes version:
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): RKE v1.29.6+rke2r1
* Other configuration options or runtime services (istio, etc.): Cilium
* vault-secrets-operator version: 0.7.1
**Additional context**
We are moving to the latest available version at this moment (0.8.1), but there are no references on the changelog for metrics or observability that would indicate this being fixed or improved.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the controller_resource_status metric instrumentation for the vaultconnection and vaultauth controllers, then reproduce the reported connectivity loss by blocking access to Vault. Confirm how the metric is updated on failure and recovery; done means it changes from 1 to 0 when connectivity is lost, with a regression test covering that transition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100