Observability & alerting: populate Conditions, Prometheus metrics, Events, ServiceMonitor + alerts

Summary

Export the data the status loop already computes. reconcileJobStatus tracks per-run phases, counters, and failure logs — but observability stops at kubectl describe. This is the lowest-effort step toward a fleet backup control plane: populate Conditions, emit metrics + Events, and ship ready-made alerts.

Gaps (grounded in current code)

1. Status.Conditions declared but never written. BackupStatus.Conditions []metav1.Condition exists (api/v1/backup_types.go:129) but no reconcile path calls apimeta.SetStatusCondition. So kubectl wait --for=condition=Ready backup/... and GitOps health checks silently no-op. (Also in .planning/codebase/CONCERNS.md.)
2. No Prometheus metrics. No way to answer "which databases haven't had a successful backup in 24h?" across a fleet.
3. No Events. Failures live only in logs + status; kubectl get events shows nothing.

Proposed work

O1 — Populate Conditions

In reconcileJobStatus, set standard Conditions via apimeta.SetStatusCondition with observedGeneration:

• Scheduled (CronJob exists/matches spec), Ready (last run succeeded / False on failure), Degraded (≥N consecutive failures), BackupVerified (when verification lands). Unblocks kubectl wait and Argo/Flux health.

O2 — Prometheus metrics

Register collectors with the controller-runtime metrics registry, labeled namespace + backup:

gobackup_backup_last_success_timestamp_seconds{namespace,backup}   gauge
gobackup_backup_last_run_timestamp_seconds{namespace,backup}       gauge
gobackup_backup_runs_total{namespace,backup,phase}                 counter
gobackup_backup_consecutive_failures{namespace,backup}             gauge
gobackup_backup_duration_seconds{namespace,backup}                 histogram

Values already exist in reconcileJobStatus (SuccessCount, FailureCount, timestamps, phase). The operator already exposes a metrics endpoint (config/prometheus), so this is mostly wiring. Reset a Backup's series on deletion (finalizer/GC) to avoid stale gauges.

O3 — Events

Inject record.EventRecorder and emit on transitions: BackupSucceeded, BackupFailed, CronJobRecreated, ConfigInvalid.

O4 — Ship the alerting (helm, values-gated)

• ServiceMonitor/PodMonitor for the operator metrics endpoint.
• PrometheusRule: GoBackupBackupFailed (consecutive_failures ≥1 for >15m), GoBackupBackupTooOld (now − last_success > 26h), GoBackupNoRecentRun (scheduled but no run in 2× interval).
• Starter Grafana dashboard (fleet table: backup, last-success age, failures, last duration).

Phases

1. O1 Conditions + O3 Events — no new infra, unblocks kubectl wait + visibility.
2. O2 metrics + series GC on delete.
3. O4 ServiceMonitor + PrometheusRule + dashboard (off by default).

Risks

• Cardinality: labels limited to namespace + backup (+ bounded phase); never per-run/job.
• Stale series: must reset metrics on Backup deletion (ties to missing-finalizer item in CONCERNS).
• Condition churn: only write on real transitions (reuse the existing same-phase guard).

Success criteria

• kubectl wait --for=condition=Ready backup/x works.
• A Prometheus query lists every Backup whose last success is >24h old, cluster-wide.
• Failures appear as Events and fire a default alert within 15 minutes.
• Deleting a Backup removes its metric series.

Start with reconcileJobStatus and BackupStatus.Conditions in api/v1/backup_types.go, then inspect the existing metrics endpoint under config/prometheus. Review how the operator handles status transitions, deletion, and controller-runtime wiring before separating the Conditions, Events, metrics, and Helm alerting phases. Done means the listed conditions, events, metrics, series cleanup, and values-gated monitoring resources meet the stated success criteria.