Kuadrant / Kuadrant/kuadrant-operator
PersistentErrorTracker prematurely marks errors "resolved" when an unrelated event triggers reconciliation
- Dominant language
- Go
- Stars
- 94
- Forks
- 89
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 43
Description
**Describe the bug**
The non-blocking error handling introduced in #2043 can mark an error as `resolved` even though the underlying operation is still failing. `PersistentErrorTracker.UpdateFromRegistry` treats an error as resolved whenever it's absent from a reconciliation cycle's `ErrorRegistry`. But a workflow run only exercises the reconcilers whose subscription matches the triggering event, so an error recorded by reconciler A gets cleared by a workflow triggered by an unrelated event B, where A never ran and never re-recorded its still-failing error. The retry loop can therefore give up before the underlying condition clears, and the error is dropped from the tracker without having actually been resolved.
Surfaced while validating #2088 (fix for #1578).
**To Reproduce**
Steps to reproduce the behavior:
1. Deploy the operator to a cluster (`make local-setup`).
2. Remove the `create` verb from the `authorinos` rule in the `kuadrant-operator-manager-role` ClusterRole, so the operator can no longer create Authorino resources.
```bash
# strip "create" from the authorinos rule in the operator's ClusterRole
kubectl get clusterrole kuadrant-operator-manager-role -o json | \
jq --arg idx "$(kubectl get clusterrole kuadrant-operator-manager-role -o json | jq '.rules | map(.resources // [] | contains(["authorinos"])) | index(true)')" \
'.rules[$idx|tonumber].verbs |= map(select(. != "create"))' | kubectl apply -f -
```
3. Create a `Kuadrant` CR (this also generates unrelated Limitador reconcile events):
```bash
kubectl apply -f - <<'EOF'
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
name: kuadrant
namespace: kuadrant-system
spec: {}
EOF
```
4. Watch the operator logs, do not restore the RBAC. The Authorino create keeps failing, yet the error is marked `resolved` in between failures:
```json
{"level":"error","ts":"2026-07-10T09:00:05.232Z","logger":"kuadrant-operator.AuthorinoReconciler","msg":"failed to create authorino resource","error":"...forbidden..."}
{"level":"info","ts":"2026-07-10T09:00:05.310Z","logger":"kuadrant-operator.PersistentErrorTracker","msg":"non-blocking error resolved","key":"Authorino.operator.authorino.kuadrant.io/kuadrant-system/authorino/create","attempts":2}
{"level":"error","ts":"2026-07-10T09:00:05.377Z","logger":"kuadrant-operator.AuthorinoReconciler","msg":"failed to create authorino resource","error":"...forbidden..."}
{"level":"error","ts":"2026-07-10T09:00:07.387Z","logger":"kuadrant-operator.AuthorinoReconciler","msg":"failed to create authorino resource","error":"...forbidden..."}
{"level":"info","ts":"2026-07-10T09:00:07.465Z","logger":"kuadrant-operator.PersistentErrorTracker","msg":"non-blocking error resolved","key":"Authorino.operator.authorino.kuadrant.io/kuadrant-system/authorino/create","attempts":2}
```
**Expected behavior**
An error should only be marked `resolved` when the operation that recorded it actually succeeds (or is genuinely no longer needed) not merely because a reconcile cycle triggered by an unrelated event didn't re-record it. Errors whose recording reconciler didn't run in a given cycle should be left untouched, so retries continue until the underlying condition actually clears.
**Additional context**
The `non-blocking error resolved` line in the reproduction is emitted by `PersistentErrorTracker.UpdateFromRegistry` in `internal/controller/reconciliation_errors.go`, which removes any tracked error that is not present in the current reconciliation cycle's `ErrorRegistry`.
This is shared error-handling code introduced in #2043; it is not specific to the Authorino or Limitador reconcilers. Because the resolution logic operates on the shared tracker, it applies to any reconciler that records errors into it (e.g. `AuthConfigsReconciler` and `LimitadorLimitsReconciler`, also wired in #2043).
The reproduction uses the Authorino create path (from #2088) only because it is the simplest way to inject a persistent, easily-triggered failure.
Contributor guide
Research direction
Start in internal/controller/reconciliation_errors.go at PersistentErrorTracker.UpdateFromRegistry, then trace how reconcilers populate the shared ErrorRegistry. Reproduce the unrelated-event scenario described in the issue and verify that errors from reconcilers that did not run remain tracked. Done means errors are resolved only after their operation succeeds or is no longer needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100