ADORSYS-GIS / ADORSYS-GIS/lightbridge-authz

[Story]: Three auth alerts, one per incident class

Open
#641 0 comments 0 reactions 0 assignees View on GitHub
user-story
Dominant language
Rust
Stars
0
Forks
1
Avg merge
6h 42m
Merged PRs (30d)
246

Description

## User Story

**As** whoever is on call,
**I want** to be told when authentication breaks,
**so that** I stop learning about it from a user who cannot work.

## Source of truth (links)

Parent epic #638. Each alert below is derived from one real incident: #623, #626, and the 2026-09-02 ArgoCD deadlock (ADR-0031).

## Context

All three incidents were found by a human noticing. Time-to-detect was 3h32m, days, and "got lucky while watching an unrelated rollout".

## Acceptance Criteria

Three alerts, each mapped to an incident class and each pointing at a runbook section.

- [ ] **A1 — fresh logins collapse while refreshes stay healthy.** The #623 signature. Deliberately a *ratio*, not an absolute: raw login volume is spiky and diurnal, and "refreshes healthy" is what distinguishes a broken RP leg from a quiet night. Firing on absolute login count would have been noise every night and muted before it ever mattered.
- [ ] **A2 — any `rp leg failure`.** Rare in normal operation and always worth a look; #623 emitted these continuously for 3h32m. If it turns out to be noisier than expected, the fix is to scope the reasons, not to raise the threshold until it is silent.
- [ ] **A3 — ArgoCD `lightbridge-app` unhealthy, OR `Synced` with a stale `operationState.finishedAt`.** The second half is the important one: ai-helm ADR-0135 documented ArgoCD reporting `Synced` off cached manifests while applying nothing, and on 2026-09-02 the app claimed to be synced at a revision whose sync operation had finished **35 minutes before that commit existed**. An alert on health alone would have missed it.
- [ ] Every alert links to `docs/runbooks/signing-key-management.md` or `auth-flows.md` §3a. **An alert with no runbook does not ship.**
- [ ] Thresholds justified against real traffic, with the query and the observed baseline recorded on the ticket.

## Out of Scope

Paging policy and routing. Alerts on GenAI usage/cost (#581).

## Dependencies

Story 2 (#640) — alerts should reuse the panels' queries so the two cannot drift apart.

## Risks

**Alert fatigue is the failure mode.** A noisy auth alert gets muted and is then strictly worse than no alert, because it produces false confidence. Prefer three alerts that always mean something over ten that mostly do not. If A2 proves chatty, narrow the reasons rather than raising the threshold until it stops firing.

## Test Plan / Verification evidence

For each alert, induce the condition in staging and show it fired:
1. A1 — break discovery; fresh logins collapse, refreshes flat, alert fires.
2. A2 — force an RP-leg failure.
3. A3 — leave the app `Degraded`; separately, confirm the stale-`operationState` half fires on a false green.

Record time-to-fire for each. The epic's target is **under 5 minutes**.

## AI Usage Declaration

AI (Claude Opus 5, Claude Code) diagnosed all three incidents and drafted this story.

- [x] Each alert is derived from a specific incident with a known signature, not from a generic checklist.
- [x] A3's stale-`operationState` condition comes from a real observation: on 2026-09-02 ArgoCD reported `Synced` at `b7a6b03` while its last sync operation had finished 35 minutes before that commit existed.
- [x] The alert-fatigue risk is stated as the primary failure mode, with the explicit instruction to narrow rather than desensitise.

## Human verification completed

- [ ] Owner confirms three alerts is the right number to start with.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.