ADORSYS-GIS / ADORSYS-GIS/lightbridge-authz
[Story]: Phase 7 — budget domain hardening (metrics, alerts, load tests)
- Ngôn ngữ chính
- Rust
- Star
- 0
- Fork
- 1
- Merge trung bình
- 6 giờ 42 phút
- Pull request đã merge (30 ngày)
- 246
Mô tả
### Story Statement
As **whoever is on call for this platform**,
I want **the budget domain (ledger, policy engine, workflows) instrumented with metrics and alerts, and load-tested**,
so that **a grant that silently stopped writing, a policy engine that silently started denying everything, or a review queue that silently backed up is caught by a dashboard or a page — not by a user complaint.**
### Real Intent
Everything built in #189–#191 and the phases beyond them is only as trustworthy as the platform's ability to notice when it breaks. This is the story that closes that gap. Outline only — filed now so the epic's delivery-plan PR breakdown has an issue to reference; genuinely last in sequence, since it instruments what the earlier phases build.
### Background and Context
The three runbooks already committed to this repo (`docs/runbooks/roll-back-a-budget-policy.md`, `stuck-augmentation-request.md`, `budget-tier-rekey-cutover.md`) each lean on specific signals — `/health`'s `activePolicyRevision`/`lastLoadError`, a `policy_effect` breakdown query, the review-queue backlog, `gateway_ratelimit_spend_micro_usd` — that this story should make into first-class metrics/alerts rather than ad hoc queries an operator has to remember to run.
### Source of truth (links)
- Epic: #188 — success metrics table (manual edits per refill, grants without an audit record, time from request to granted budget, policy changes requiring a deploy)
- Runbooks: docs/runbooks/roll-back-a-budget-policy.md, docs/runbooks/stuck-augmentation-request.md, docs/runbooks/budget-tier-rekey-cutover.md
- Load-test precedent: `just load-test` / the Goose framework already used for the OPA endpoint (see AGENTS.md "Load Tests")
### Acceptance Criteria
#### Functional
- [ ] Metrics exist for: grants issued (by source), requests denied (by reason), requests sent to manual review, review-queue depth and age, policy activation/rollback events, and policy-engine evaluation latency/timeouts.
- [ ] Alerts exist for at least: a review queue growing unboundedly with no reviewer action, a policy load failure, and the fail-closed paths (unknown claim, bad rule data, evaluation timeout) firing above a baseline rate.
- [ ] Load tests exist for the refill request path, following the existing Goose pattern (`lightbridge-authz-rest`'s `load-tests` feature), and report findings the way the existing OPA load test does (requests/sec, latency).
#### Negative / Edge Cases
- [ ] Given the review-queue-depth alert threshold, when it fires, then the stuck-augmentation-request runbook's §2 diagnostic (does anyone hold `budget:review` at all) is the documented first response.
- [ ] Given a policy rollback per the roll-back runbook, when it happens, then it is itself an alertable/observable event, not just a manual `curl`.
#### Non-Functional
- [ ] Metrics and alerts are documented (naming, thresholds, what "normal" looks like) — a metric nobody knows how to read is not an improvement over no metric.
- [ ] `cargo clippy --all-targets --all-features -- -D warnings` clean.
### Out of Scope
- Any new functional behavior in the ledger, policy engine, or workflows — this story observes what #189–#191 and Phases 2b/5/6b built, it does not change their behavior.
- Gateway-side dashboards (the `prometheus-redis-exporter` / `ratelimit_quota.py` co-change) — that is ai-helm's side of Phase 6a, already called out as its own risk in the epic.
### Dependencies and Blockers
- Depends on #189, #190, #191, and realistically Phases 2b/5/6b, since it instruments all of them — this is genuinely the last phase to implement, even though it is filed now.
### Assumptions
1. The platform's existing OTEL/Jaeger instrumentation (AGENTS.md "Observability") is the substrate these new metrics attach to — this story extends it, it does not introduce a new observability stack.
### Implementation Notes
Guidance, not gospel: mine the three existing runbooks for every ad hoc query they already tell an operator to run by hand, and turn each into either a dashboard panel or an alert — that list is close to the actual requirements, because it was written by someone imagining debugging this system at 2am before any of it existed.
### Test Expectations
- Load test results committed/reported the same way the existing OPA load test's findings are recorded (AGENTS.md: "~600-1000 requests per second... minimal latency").
- A test/demonstration that each alert actually fires under the condition it claims to detect (not just that the metric exists).
### Verification evidence
*To be completed on delivery.*
### Human accountable owner
@stephane-segning — filed at outline depth on the epic owner's behalf per the #188 backlog comment (2026-08-01); needs their confirmation before entering sprint planning.
### AI Usage Declaration
Drafting this story
### Human verification completed
- [ ] I checked the story against the source of truth
- [ ] I confirmed the acceptance criteria
- [ ] I am the accountable owner and accept responsibility for this story.
> Filed by an AI planning session as part of decomposing epic #188 (see the delivery plan discussion on that issue). None of the boxes above are ticked because the session that filed this is not the accountable owner — @stephane-segning, please review and tick what you've verified.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.