HarperFast / HarperFast/harper-pro
Monitoring & status improvements (tracking)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Why
We ship ~333 GB/day of logs to Datadog (2.33 TB in the audited week, 99.4% from `harperdb`), and the entire alerting signal extracted from that stream is **two literal strings** (the restart-detector log monitors). Harper exposes **zero application metrics**. Real incidents (replication auth-flap running for weeks at ~10M log events/day, silent blob divergences: #386, #426, #502) produced millions of log lines and no page — while ~87% of "error"-status events in Datadog aren't errors at all (stderr mis-tagging + multi-line fragmentation). Full audit numbers and findings: design doc (link below).
## Architecture (four planes)
1. **Status plane** — problems as durable, queryable, alertable *state* with explicit resolution and remediation, not log inference. Foundation: PR HarperFast/harper#372, re-landed in phases.
2. **Metrics plane** — incremental in-process collectors exposed via a native OpenMetrics endpoint, scraped by the Grafana Alloy collectors already on Fabric hosts into Mimir (the same stack the migration's alert rules run on); high-cardinality diagnostic series stay in Harper, queried in place via the published [grafana-datasource](https://github.com/HarperFast/grafana-datasource) plugin.
3. **Fleet plane** — status tables aggregated across clusters via Harper's own replication/real-time layer; alert on status *transitions* (audience=operator, level=error), push not poll.
4. **Log plane** — demoted to forensics: structured JSON, deduped, audience-tagged, with only operator-audience warn+ shipped.
## Sub-issues
**Status plane**
- [ ] HarperFast/harper#1639 — re-land PR #372 in phases (+ audience, remediation)
- [ ] HarperFast/harper#1638 — `audience: operator | user` on logs + status
**Metrics plane**
- [ ] HarperFast/harper#1636 — native OpenMetrics `/metrics` endpoint (retires prometheus-exporter polling; subsumes HarperFast/harper#376, #592, #961, #682, #668)
**Log plane**
- [ ] HarperFast/harper#1637 — structured JSON emission, multi-line-safe, ANSI stripping
- [ ] #262 — dedup/run-counts for repeated entries (existing)
- [ ] HarperFast/host-manager#134 — agent-side pipeline: multi-line aggregation, level remapping, masking, filtering (interim mitigations + cost lever)
**Alerting / inventory**
- [ ] HarperFast/central-manager#417 — monitor inventory hygiene + decommission reaping
**Replication observability** (tracked under #437, Replication W8):
- HarperFast/harper#1635 — auth-failure backoff + per-peer status entry
- #531 — live auth-flap incident on multiple production clusters (see #531 Customer Impact field)
## Immediate cost levers (independent of the redesign)
fix #531 (~36% of indexed volume), host-manager#134 items 1–4, central-manager#417 sweep, cookie/PEM redaction (compliance — in flight separately).
## Grafana migration context
The Datadog→Grafana migration ([HarperFast/observability](https://github.com/HarperFast/observability) — Grafana Cloud IaC via Terraform; 25 alert rules live-but-muted against Mimir, synthetic checks replacing Datadog synthetics, D25/D26 division with central-manager) is the **infrastructure half** of this picture: it replaces the monitoring backend but doesn't create application-level signal. This tracking issue is the application half — the status/metrics/log planes give the new backend something worth alerting on. Consequences for sub-issues: host-manager#134 is explicitly interim (scoped by the Datadog log-cutover timeline; masking immediately regardless), and central-manager#417's systematic pieces should be built in the Grafana SM provisioning path, not the Datadog code.
## Related
#437 (Replication W8 — the replication-plane slice of this), #430, PR HarperFast/harper#372, HarperFast/observability, HarperFast/grafana-datasource, design doc: [Holistic Fleet Monitoring & Status Overhaul — Design](https://harperdb.atlassian.net/wiki/spaces/SD/pages/829423618/Holistic+Fleet+Monitoring+Status+Overhaul+Design).
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Research direction
Start with the linked Holistic Fleet Monitoring & Status Overhaul design document, then read the listed sub-issues #1635–#1639, #262, host-manager#134, and central-manager#417. No single file, test, or entry point is named here. This tracking issue is complete when its status, metrics, fleet, and log-plane work is addressed through those tracked items.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, javascript, nodejs, terraform
- Domain
- backend, devops, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100