ADORSYS-GIS / ADORSYS-GIS/lightbridge-authz

[Story]: Three auth alerts, one per incident class

Aberta
#641 0 comentários 0 reações 0 responsáveis Ver no GitHub
user-story
Linguagem predominante
Rust
Estrelas
0
Forks
1
Merge médio
7h 7min
PRs com merge (30d)
237

Descrição

## User Story

**As** whoever is on call,
**I want** to be told when authentication breaks,
**so that** I stop learning about it from a user who cannot work.

## Source of truth (links)

Parent epic #638. Each alert below is derived from one real incident: #623, #626, and the 2026-09-02 ArgoCD deadlock (ADR-0031).

## Context

All three incidents were found by a human noticing. Time-to-detect was 3h32m, days, and "got lucky while watching an unrelated rollout".

## Acceptance Criteria

Three alerts, each mapped to an incident class and each pointing at a runbook section.

- [ ] **A1 — fresh logins collapse while refreshes stay healthy.** The #623 signature. Deliberately a *ratio*, not an absolute: raw login volume is spiky and diurnal, and "refreshes healthy" is what distinguishes a broken RP leg from a quiet night. Firing on absolute login count would have been noise every night and muted before it ever mattered.
- [ ] **A2 — any `rp leg failure`.** Rare in normal operation and always worth a look; #623 emitted these continuously for 3h32m. If it turns out to be noisier than expected, the fix is to scope the reasons, not to raise the threshold until it is silent.
- [ ] **A3 — ArgoCD `lightbridge-app` unhealthy, OR `Synced` with a stale `operationState.finishedAt`.** The second half is the important one: ai-helm ADR-0135 documented ArgoCD reporting `Synced` off cached manifests while applying nothing, and on 2026-09-02 the app claimed to be synced at a revision whose sync operation had finished **35 minutes before that commit existed**. An alert on health alone would have missed it.
- [ ] Every alert links to `docs/runbooks/signing-key-management.md` or `auth-flows.md` §3a. **An alert with no runbook does not ship.**
- [ ] Thresholds justified against real traffic, with the query and the observed baseline recorded on the ticket.

## Out of Scope

Paging policy and routing. Alerts on GenAI usage/cost (#581).

## Dependencies

Story 2 (#640) — alerts should reuse the panels' queries so the two cannot drift apart.

## Risks

**Alert fatigue is the failure mode.** A noisy auth alert gets muted and is then strictly worse than no alert, because it produces false confidence. Prefer three alerts that always mean something over ten that mostly do not. If A2 proves chatty, narrow the reasons rather than raising the threshold until it stops firing.

## Test Plan / Verification evidence

For each alert, induce the condition in staging and show it fired:
1. A1 — break discovery; fresh logins collapse, refreshes flat, alert fires.
2. A2 — force an RP-leg failure.
3. A3 — leave the app `Degraded`; separately, confirm the stale-`operationState` half fires on a false green.

Record time-to-fire for each. The epic's target is **under 5 minutes**.

## AI Usage Declaration

AI (Claude Opus 5, Claude Code) diagnosed all three incidents and drafted this story.

- [x] Each alert is derived from a specific incident with a known signature, not from a generic checklist.
- [x] A3's stale-`operationState` condition comes from a real observation: on 2026-09-02 ArgoCD reported `Synced` at `b7a6b03` while its last sync operation had finished 35 minutes before that commit existed.
- [x] The alert-fatigue risk is stated as the primary failure mode, with the explicit instruction to narrow rather than desensitise.

## Human verification completed

- [ ] Owner confirms three alerts is the right number to start with.

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Start with parent epic #638 and Story 2 #640, then inspect the panels' queries and the alert configuration they are meant to reuse. Read docs/runbooks/signing-key-management.md and auth-flows.md §3a, and review incidents #623, #626, and ADR-0031. Done means three justified alerts link to a runbook and each condition fires in staging within the epic's under-five-minute target.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Domínio
authentication, infrastructure, observability
Tipo de issue
Funcionalidade
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Ativa
Clareza
Razoavelmente clara
Facilidade para iniciantes
45/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.