[Epic]: Nothing watches whether authentication works — auth-health observability on the existing Alloy/Grafana stack

## Executive Summary

Nothing watches whether **authentication itself is working**. Prod has Grafana, Loki, Tempo, Mimir and 4 Alloy agents already collecting `lightbridge-authz` logs — and **32 dashboards, none of them about auth**. Every dashboard is Envoy AI Gateway usage/cost. The two whose names match "token" are about GenAI spend.

The consequence, measured: on 2026-09-01 a config change broke **100% of fresh logins for 3h32m** (14:06Z → 17:38Z) and nothing detected it. It was found because a human tried to log in.

## Strategic Intent

Make an authentication outage visible in **minutes, from a dashboard**, instead of hours, from a user complaint — using observability infrastructure that is already deployed and already receiving our logs.

## Problem Statement

Three production incidents in ~36 hours, none detected by tooling:

| when | incident | how it was found | duration |
|---|---|---|---|
| 2026-09-01 14:06→17:38 | `oauth2.jwks_url` repointed; `discover()` failed; every fresh login broke (#623) | user reported "cannot log in" | **3h32m** |
| 2026-09-02 ~00:11 | browser logout cascaded into every CLI's refresh chain (#626) | user reported repeated re-logins | days, intermittent |
| 2026-09-02 03:0x | ArgoCD deadlock: mcp at sync-wave 0 ahead of its migration | noticed while watching an unrelated rollout | ~15m, lucky |

Three properties made all three hard:

1. **The signal existed and nobody looked.** Incident 1's signature was unmistakable in Loki the whole time: `rp leg failure reason="begin_device_failed"` repeating while `token-exchange refreshed access token` stayed healthy. *Fresh logins broken, refreshes fine* — a two-series graph.
2. **Refresh failures were silent.** Five distinct causes returned one indistinguishable `400 invalid_grant` and only the reuse cascade logged. #637 has just fixed this by adding a `reason=` field to every refusal — **so the data now exists and there is still nowhere to see it.**
3. **ArgoCD can report `Synced` while applying nothing** (ai-helm ADR-0135), so the deploy surface can lie in the direction of "fine".

## Desired Outcome

An on-call engineer opens one dashboard and answers, in under a minute: *are people able to log in right now, and if not, why?* — and does not need to be the person who wrote the code.

## Scope (In / Out)

**In**
- Export `lightbridge-authz` traces/metrics to the **existing Alloy DaemonSet** OTLP receiver.
- One Grafana dashboard for auth health, from data already in Loki.
- Three alerts, each mapped to one incident above.

**Out**
- New observability infrastructure. Grafana, Loki, Tempo, Mimir and Alloy are all deployed; Alloy already listens on OTLP `:4317`/`:4318` and fans out traces→Tempo, logs→Loki, metrics→Mimir. **We use Alloy, not a dedicated OTel collector** (decision by the repo owner) — the other services' per-app collectors are not the pattern to copy here.
- GenAI usage/cost analytics (#581, ADR-0027/0028) — a different axis, already covered.
- Log retention/cost tuning.

## Source of truth (links)

- #623 — the 3h32m fresh-login outage
- #626 — the logout over-cascade
- #637 — the `reason=` refusal logging that makes the dashboard's most useful panel possible
- ADR-0031 — the deploy-ordering class behind incident 3
- `docs/architecture/auth-flows.md` §3a — the state machine the `reason=` values map onto
- `docs/runbooks/signing-key-management.md` — the manual procedure a dashboard should make unnecessary

## Stakeholders

Whoever is on call for `auth.ai.camer.digital`; every CLI/console user, who currently learns about auth outages by being unable to work.

## Key Assumptions

- Alloy's existing OTLP receiver is reachable from the `converse` namespace. **To verify in story 1** — assumed, not confirmed.
- `lightbridge-authz` logs are already in Loki via Alloy's pod-log scraping (its `loki.write` is configured). To confirm with a real query before building panels on it.
- The `reason=` field from #637 reaches Loki as a parseable logfmt field.

## Constraints

- Alloy only, no new collector.
- Panels must come from data already emitted; anything needing new instrumentation is a separate ticket, not a silent scope expansion.
- Alerts must be actionable — each maps to a runbook section or it does not ship.

## Risks

- **Alert fatigue.** A noisy "auth is broken" alert gets muted and is then worse than nothing. Each alert needs a threshold justified against real traffic, not a guess.
- **The most valuable panel depends on #637 being deployed.** Until it is, `reason=` is absent and that panel is empty.
- **Over-scoping into product analytics.** This is operational health only; if it starts answering "who used what", it has drifted into #581.

## Success metrics

- An induced fresh-login failure in staging is visible on the dashboard within **2 minutes** and fires an alert within **5**.
- Each of the three incidents above, replayed against the dashboard, is diagnosable **without reading source**.
- Time-to-detect for an auth outage goes from *"a user tells us"* to *"an alert fires"*.

## Child User Stories

1. Export `lightbridge-authz` telemetry to the existing Alloy OTLP receiver
2. An auth-health Grafana dashboard
3. Three alerts, one per incident class

## Human accountable owner

@stephane-segning

## AI Usage Declaration

AI (Claude Opus 5, Claude Code) diagnosed all three incidents against the running cluster, surveyed the deployed observability stack, and drafted this epic.

- [x] Incident durations and signatures are from real pod logs and the live ArgoCD state, not reconstructed.
- [x] The claim "32 dashboards, none about auth" was verified by listing `GrafanaDashboard` CRs in `hetzner-prod`.
- [x] Alloy's existing OTLP receiver was confirmed by reading its live ConfigMap, which is why story 1 is a configuration change rather than a deployment.

## Human verification completed

- [ ] Owner confirms the three-story split and the Alloy-only constraint.