ADORSYS-GIS / ADORSYS-GIS/lightbridge-authz

[Story]: Phase 7 — budget domain hardening (metrics, alerts, load tests)

Đang mở
#197 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
governance user-story
Ngôn ngữ chính
Rust
Star
0
Fork
1
Merge trung bình
6 giờ 42 phút
Pull request đã merge (30 ngày)
246

Mô tả

### Story Statement

As **whoever is on call for this platform**,
I want **the budget domain (ledger, policy engine, workflows) instrumented with metrics and alerts, and load-tested**,
so that **a grant that silently stopped writing, a policy engine that silently started denying everything, or a review queue that silently backed up is caught by a dashboard or a page — not by a user complaint.**

### Real Intent

Everything built in #189–#191 and the phases beyond them is only as trustworthy as the platform's ability to notice when it breaks. This is the story that closes that gap. Outline only — filed now so the epic's delivery-plan PR breakdown has an issue to reference; genuinely last in sequence, since it instruments what the earlier phases build.

### Background and Context

The three runbooks already committed to this repo (`docs/runbooks/roll-back-a-budget-policy.md`, `stuck-augmentation-request.md`, `budget-tier-rekey-cutover.md`) each lean on specific signals — `/health`'s `activePolicyRevision`/`lastLoadError`, a `policy_effect` breakdown query, the review-queue backlog, `gateway_ratelimit_spend_micro_usd` — that this story should make into first-class metrics/alerts rather than ad hoc queries an operator has to remember to run.

### Source of truth (links)

- Epic: #188 — success metrics table (manual edits per refill, grants without an audit record, time from request to granted budget, policy changes requiring a deploy)
- Runbooks: docs/runbooks/roll-back-a-budget-policy.md, docs/runbooks/stuck-augmentation-request.md, docs/runbooks/budget-tier-rekey-cutover.md
- Load-test precedent: `just load-test` / the Goose framework already used for the OPA endpoint (see AGENTS.md "Load Tests")

### Acceptance Criteria

#### Functional
- [ ] Metrics exist for: grants issued (by source), requests denied (by reason), requests sent to manual review, review-queue depth and age, policy activation/rollback events, and policy-engine evaluation latency/timeouts.
- [ ] Alerts exist for at least: a review queue growing unboundedly with no reviewer action, a policy load failure, and the fail-closed paths (unknown claim, bad rule data, evaluation timeout) firing above a baseline rate.
- [ ] Load tests exist for the refill request path, following the existing Goose pattern (`lightbridge-authz-rest`'s `load-tests` feature), and report findings the way the existing OPA load test does (requests/sec, latency).

#### Negative / Edge Cases
- [ ] Given the review-queue-depth alert threshold, when it fires, then the stuck-augmentation-request runbook's §2 diagnostic (does anyone hold `budget:review` at all) is the documented first response.
- [ ] Given a policy rollback per the roll-back runbook, when it happens, then it is itself an alertable/observable event, not just a manual `curl`.

#### Non-Functional
- [ ] Metrics and alerts are documented (naming, thresholds, what "normal" looks like) — a metric nobody knows how to read is not an improvement over no metric.
- [ ] `cargo clippy --all-targets --all-features -- -D warnings` clean.

### Out of Scope

- Any new functional behavior in the ledger, policy engine, or workflows — this story observes what #189–#191 and Phases 2b/5/6b built, it does not change their behavior.
- Gateway-side dashboards (the `prometheus-redis-exporter` / `ratelimit_quota.py` co-change) — that is ai-helm's side of Phase 6a, already called out as its own risk in the epic.

### Dependencies and Blockers

- Depends on #189, #190, #191, and realistically Phases 2b/5/6b, since it instruments all of them — this is genuinely the last phase to implement, even though it is filed now.

### Assumptions

1. The platform's existing OTEL/Jaeger instrumentation (AGENTS.md "Observability") is the substrate these new metrics attach to — this story extends it, it does not introduce a new observability stack.

### Implementation Notes

Guidance, not gospel: mine the three existing runbooks for every ad hoc query they already tell an operator to run by hand, and turn each into either a dashboard panel or an alert — that list is close to the actual requirements, because it was written by someone imagining debugging this system at 2am before any of it existed.

### Test Expectations

- Load test results committed/reported the same way the existing OPA load test's findings are recorded (AGENTS.md: "~600-1000 requests per second... minimal latency").
- A test/demonstration that each alert actually fires under the condition it claims to detect (not just that the metric exists).

### Verification evidence

*To be completed on delivery.*

### Human accountable owner

@stephane-segning — filed at outline depth on the epic owner's behalf per the #188 backlog comment (2026-08-01); needs their confirmation before entering sprint planning.

### AI Usage Declaration

Drafting this story

### Human verification completed

- [ ] I checked the story against the source of truth
- [ ] I confirmed the acceptance criteria
- [ ] I am the accountable owner and accept responsibility for this story.

> Filed by an AI planning session as part of decomposing epic #188 (see the delivery plan discussion on that issue). None of the boxes above are ticked because the session that filed this is not the accountable owner — @stephane-segning, please review and tick what you've verified.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.