HarperFast / HarperFast/harper-pro

Replication W8: Observability & metrics pipeline

Open
#437 1 comment 0 reactions 0 assignees View on GitHub
area:metrics area:replication enhancement
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

**Workstream W8 of #430 · observability & metrics pipeline**

## Summary
Today there are *two half-pipes that don't meet*: `cluster_status` (an ephemeral poll snapshot of health gauges) and `hdb_analytics` (durable time-series, but only byte counters). Neither gives an operator a divergence alarm — which is why #426 (silent ~39% loss) and #386 (344 lost attachments) went undetected with `connected:true`. Connect the pipes and add the missing signals.

## Tier 1 — extend `cluster_status` (cheap, high diagnostic value, mostly already filed)
- [ ] **Last connection error {message, code}** per socket (#214) — store the error the close/error handlers already see. Biggest "why is it down" win.
- [ ] **Resume-cursor + applied-vs-head lag** per socket — surface `lastDurableSequenceId`/persisted `seqId` and a computed lag; **fix `lastReceivedVersion` not populating on the relay/`data` path** (it's a non-signal exactly when you need it — see #426).
- [ ] **Removed-node truth** + replace the hardcoded `is_enabled:true` with real membership (#217).
- [ ] **Shard count** (#204).
- [ ] **Blob in-flight depth + "cursor held by blob gap" flag** (complements the already-shipped `blobReplicationFailures`).
- [ ] **Audit/transaction-log backlog depth** (pending between cursor and head). (Reframes the abandoned-NATS #261 onto the live audit log.)
- [ ] Make the op cheap — drop the full `getSystemInfo` (#260).

## Tier 2 — a real metrics pipeline (trends + alerts + divergence)
- [ ] **Promote replication-health gauges into `hdb_analytics`** (it already ingests replication *bytes* — extend to lag, back-pressure %, blob-failure rate, connection state, cursor position).
- [ ] **Prometheus / OpenTelemetry export** over that series (none exists today).
- [ ] **Divergence detection** — periodic cross-peer record/blob-count comparison emitting an alertable metric. This is the connective tissue with W2: it converts every "silent divergence" into a loud, actionable signal *before* data loss.

## Tier 3 — log hygiene
- [ ] **Rate-limit/dedup repeated log lines with run-counts** (#262) — defuses the #327/#349 storm floods and makes logs alert-parseable.

## Retires / advances
- [ ] #214, #217, #204, #260, #261 (reframed), #262
- **Closes the loop W2 opens** (divergence alarm)

## Dependencies
**W1** (registry/last-error surfacing), **W2** (gap signal), **W4** (per-origin lag). Tier 1 can largely proceed in Phase 1 as quick wins; Tier 2 is the strategic piece.

## Effort / risk
**M / low.**

## Acceptance criteria
- An operator can see per-subscription lag, last connection error, and cursor position.
- A diverging cluster raises an alert before data loss.
- Storm-condition logs are bounded and parseable.

---
🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing cluster_status health-gauge path and hdb_analytics replication-byte ingestion, then review dependencies W1, W2, and W4. The work is done when operators can see per-subscription lag, the last connection error, and cursor position, receive an alert before divergence causes data loss, and see bounded, parseable storm-condition logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, prometheus
Domain
databases, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.