HarperFast / HarperFast/harper-pro

Replication observability: inbound-link awareness / half-open link self-report

Open
#589 0 comments 0 reactions 0 assignees View on GitHub
area:replication enhancement
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

**Replication observability gap — surfaced by Chris Nelson (monitoring), 2026-07-15.**
Child of #430 (Replication epic) / #532 (Monitoring tracking); complements #431 (W1 connection truth) and #437 (W8 observability).

## Problem
Connection health is tracked **outbound-only**. A node reports its *own* outbound sockets as connected, but has no model of which peers are **expected to connect inbound** to it, nor whether they actually are. Half-open links dominate real failures and this makes them provably undetectable in the worst case.

Live example (ford prod): **`us-ord-1` reports 7/7 connected while all 7 peers report they cannot reach it.** Today we infer the culprit from the *accusation pattern* across peers — which is provably blind for a 2-node cluster (each side sees one accusation, no majority). If a node tracked **expected-vs-actual inbound** peers, the sick node would self-report instead of requiring cross-node inference.

This is the same structural idea Chris calls "subscription topology as data": *expected AND not flowing = alert*, the mirror of the CM desired-state feed.

## Why #431 doesn't cover this
#431 (W1) makes the *outbound* `connected` bit level-triggered and truthful (liveness watchdog owns it). That fixes "my socket died without a `close`" — it does not give a node a notion of its **inbound** expectations. `us-ord-1`'s outbound watchdog can be perfectly healthy while every inbound link is dead.

## Design direction
- Derive per-node **expected inbound peer set** from subscription topology / membership (the set of nodes that should hold an open link *to* this node).
- Track **actual inbound** links (peer node id per accepted replication socket) in the same shared-memory registry #431 introduces.
- Expose `expected_inbound` vs `actual_inbound` per node in `cluster_status` and as a scrapable metric (per-peer labels) under #437 Tier 2 / harper#1636.
- Alert rule: `expected_inbound - actual_inbound != ∅` on the node that is *missing* the link ⇒ the sick node self-reports, no accusation-pattern inference, works at N=2.

## Acceptance criteria
- A node whose inbound links are all dead self-reports the deficit (does not require peers to accuse it).
- Works correctly on a 2-node cluster.
- `expected` vs `actual` inbound topology is queryable and scrapable per peer.

---
🤖 Filed by KrAIs on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start with the shared-memory registry described in #431 and the cluster_status entry point, then trace subscription topology and membership to determine expected inbound peers. Done means a node exposes expected versus actual inbound peers in cluster_status and a per-peer scrapable metric, including correct self-reporting for a two-node cluster.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
distributed-systems, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.