VictoriaMetrics / VictoriaMetrics/operator
Runtime failure signals
@AquaOctet is already working on this.
Since Aug 21, 2026.
- Dominant language
- Go
- Stars
- 589
- Forks
- 229
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 77
Description
Change:
Pod Ready means that the Operator process is running. It does not mean that every controller is making progress.
Pod readiness is not a substitute for controller signals. We want to add the separate signals below:
- add the existing
operator_object_status{status="failed"}metric to the dashboard - add the existing
workqueue_longest_running_processor_secondsmetric to the dashboard and ship a stalled-reconcile alert - add the existing
configreloader_k8s_watch_errors_totalandconfigreloader_last_reload_errors_totalmetrics to the dashboard and alerts - add
operator_cache_watch_errors_total{type,reason}through controller-runtimeDefaultWatchErrorHandler - remove the stale operator_prometheus_converter_watch_events_total query from
dashboards/operator.jsonin the VictoriaMetrics repository
Do not alert only because reconcile count is zero. An idle controller is valid.
Note: that pod readiness is itself currently unreliable: the chart's
readinessProbetargets/health, a no-op endpoint, so readiness today reflects only that the HTTP server is listening.
Done when:
The NetworkPolicy reproduction fires a stalled-reconcile or cache-watch alert while the pod remains Ready. The config-reloader permission test fires a sidecar alert. A failed CR appears in the object-status panel. Healthy idle controllers do not alert.
Product result:
Operators see confirmed reconcile and config-reload failures before users report missing changes.
Failure Coverage:
| Failure | Covered by | Required result |
|---|---|---|
| Package parts do not match | Tested release record; Full-package pre-release tests | The release record names the exact package. The same package passes the test. |
| API, RBAC, or cache cannot sync | Full-package pre-release tests; Upgrade dry-run check; Runtime failure signals | The test fails. Dry-run reports the exact error or timeout. Runtime monitoring alerts if it still happens. |
| A child watch is missing or stops | Full-package pre-release tests; Runtime failure signals | The test changes a child object and verifies reconcile and watch recovery. Runtime monitoring reports a watch error. |
| Upgrade restarts a managed workload unexpectedly | Full-package pre-release tests | The test fails on an unexpected restart or availability loss. |
| Config-reloader cannot read or apply a Secret | Full-package pre-release tests; Runtime failure signals | The test updates the Secret and verifies reload. Runtime monitoring reports watch or reload errors. |
| Rollback fails | Full-package pre-release tests | Rollback and health checks must pass. |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.