hashicorp / hashicorp/vault-secrets-operator

VSO controller_resource_status metrics not updating

Open
#932 1 comment 2 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
600
Forks
146
Avg merge
3d 8h
Merged PRs (30d)
6

Description

**Describe the bug**

The `controller_resource_status` metrics, which contain metrics on the auth and connections status from the VSO to a Vault instance do not get updated.

Metrics for `controller="vaultconnection"` and `controller="vaultauth"` experience this issue.

From the observed behavior, the metric only gets updated when going from a failure state to a working state, the other way around doesn't happen.

**To Reproduce**

Steps to reproduce the behavior:
1. Deploy the Vault Secrets Operator to the Kubernetes cluster with proper connection configurations in place, like network policies;
2. Wait for the VSO to start, run the metric related checks and start exposing the metrics;
3. Validate that the `controller_resource_status` metrics show a `1` value, meaning that the VSO was able to connect to Vault;
4. Drop the network policies (or network connectivity) to Vault;
5. Watch the metrics, and see that they are not being update to a `0` value, meaning that the connection is failing.

Here is a screenshot of a dashboard to visualize the behavior

![Screenshot 2024-09-24 at 11 17 47](https://github.com/user-attachments/assets/9d77fedf-2a70-4af0-a3cb-43bde7954e6a)

![Screenshot 2024-09-24 at 11 27 01](https://github.com/user-attachments/assets/03d42211-4e67-4437-9771-a670fda5cbfa)

Application deployment:

There is no application involved in this case, since this is a VSO issue.

**Expected behavior**

When the connectivity to Vault becomes unavailable, the metrics should be updated to show the actual status.

**Environment**

* Kubernetes version:
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): RKE v1.29.6+rke2r1
* Other configuration options or runtime services (istio, etc.): Cilium
* vault-secrets-operator version: 0.7.1

**Additional context**

We are moving to the latest available version at this moment (0.8.1), but there are no references on the changelog for metrics or observability that would indicate this being fixed or improved.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the controller_resource_status metric instrumentation for the vaultconnection and vaultauth controllers, then reproduce the reported connectivity loss by blocking access to Vault. Confirm how the metric is updated on failure and recovery; done means it changes from 1 to 0 when connectivity is lost, with a regression test covering that transition.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.