hardbyte / hardbyte/awa

Grafana alert pack: codify the operational watch-list as importable alerts

Open
#376 0 comments 0 reactions 0 assignees View on GitHub
docs operational
Dominant language
Rust
Stars
32
Forks
5
Avg merge
15h 21m
Merged PRs (30d)
21

Description

## Motivation

The 0.5→0.6 upgrade guide and the managed-Postgres guide define a precise watch-list (ring rotation, prune blockers, queue lag, xact age), but operators must hand-build alerts from prose. `docs/grafana/` ships dashboards; it should ship alert rules too.

## Scope

Importable Prometheus/Grafana alert rules beside `docs/grafana/awa-dashboard.json`, covering at minimum:

- `pg_stat_activity.xact_start` age beyond threshold (the #169 MVCC-discipline alert promised in that issue's playbook)
- `awa_maintenance_rotate_attempts_total{awa_ring_outcome="skipped_busy"}` sustained elevation (producers outpacing consumers)
- `awa_maintenance_prune_attempts_total{awa_ring_outcome="blocked"}` non-zero (held-tx blocking reclaim)
- `awa_queue_lag_seconds` p95 vs SLO
- DLQ depth delta over window
- `awa_maintenance_branch_overrun_total` non-zero (also the telemetry gate for #303)
- backpressure signals once #341 lands

Each rule documented with meaning, likely cause, and the matching `awa doctor` check / troubleshooting entry.

## Acceptance

- Rules import cleanly into a stock Prometheus/Grafana stack; validated in CI (syntax/lint at minimum).
- Cross-referenced from the upgrade guide and operations handbook so prose watch-lists point at the shipped rules.

## References

Roadmap: `docs/0.7-roadmap.md` (WS-4, NEW-12). Related: #169 playbook, #302/#303, #341, `docs/upgrade-0.5-to-0.6.md` watch-list.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/grafana/awa-dashboard.json, the 0.5-to-0.6 upgrade guide, the managed-Postgres guide, and the referenced #169 playbook to map each watch-list item to its metric and meaning. Check how the stock Prometheus/Grafana stack validates imported rules and how CI performs syntax or lint checks. Done means the rules are importable, validated in CI, and cross-referenced from both required guides.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, postgresql, prometheus
Domain
documentation, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.