HarperFast / HarperFast/harper
False-green health surfaces: eight instances in two weeks share one design gap
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Over the last two weeks, eight separate issues have shared one shape: **a health surface reported healthy while the thing it names was broken.** Sockets connected with no data flowing. A clone Available with a fraction of its data. A component healthy with no dependencies on disk. Writes rejected on every request while the node reported fine.
Filed individually they look like eight bugs in eight subsystems. Read together they look like one design gap, because in every case the surface answers a **cheaper question than the one it appears to answer** — and the cheap question has a true answer while the real one does not.
This issue is the tracker for that gap. The individual defects should still be fixed on their own terms; what this asks is whether we should be deriving health differently, so the ninth instance doesn't have to be discovered by counting rows.
## Instances
| Issue | Surface | Said | Was |
| --- | --- | --- | --- |
| HarperFast/harper-pro#641 | `cluster_status`, backpressure, `lastReceivedRemoteTime` | all green, cursors current | ~700k of 803k writes silently skipped on three peers |
| HarperFast/harper-pro#655 | clone sync monitor | "All databases synchronized", clone marks itself `Available` | 2.2 s into a multi-GB base copy, holding a fraction of the data |
| HarperFast/harper-pro#652 | `cluster_status` | socket connected | a committed record never shipped; sender asleep until the next write |
| HarperFast/harper-pro#642 | subscription liveness (ping/pong) | ping-alive for >2 days | sender never got past pre-`DB_SCHEMA` setup |
| #1975 | `get_status` | component and all resources `healthy` | `node_modules` absent from disk; nothing could load |
| #2016 | `cluster_status`, component health | green | replication starved for days behind a corrupt frame; acknowledged writes lost |
| #2031 | `create_backup` result | success, with a `backup_id` | engine backup already purged; the id restores nothing |
| #2048 | `copy-db` exit code | `0`, "copied N entries" | copy silently corrupt and non-restorable |
Closed but the same shape, worth reading for prior art: #2001 (every write 503ing on a thread, node healthy — closed by #2007/#2009/#2050/#2052) and HarperFast/harper-pro#420.
## The four cheaper questions
Each surface substitutes a proxy for the thing it claims to measure. The proxies fall into four groups:
1. **Transport liveness for progress.** A live socket and a fresh ping prove the connection exists, not that data is moving through it. pro#641, pro#652, pro#642.
2. **Past success for present capability.** "It loaded once" is retained and reported as "it works". #1975.
3. **Vacuous truth for a met target.** When the target is missing or zero, "every target met" is trivially true. pro#655 compared against targets that were all `0` because the field they derive from is a stub (#2091) — so the check passed by having nothing to check.
4. **Local success for end-to-end success.** A step returning without throwing is reported as the operation succeeding, though later stages never ran or their failures were unawaited. #2031, #2048.
Group 3 is the one that most deserves attention: a check whose inputs are unavailable currently **passes**. That's the default that turns a missing field into a green cluster.
## The invariant worth considering
> A health surface must be derived from evidence of forward progress in the thing it names. Where that evidence is unavailable, the surface reports `unknown` or `degraded` — never `healthy`.
Two corollaries, both of which several instances above violate:
- **Vacuous checks fail closed.** No target, no watermark, no sample → not synchronized. Today, zero targets means synchronized.
- **Liveness and progress are separate fields.** Both are worth reporting; neither should be allowed to stand in for the other. A surface that folds them into one boolean will always be able to lie in one direction.
## What would settle this
I don't think we should design the fix in this issue. Two things would tell us whether it's worth a systematic pass:
1. **An inventory of health-bearing surfaces** — `cluster_status`, `get_status`, component health, clone/sync monitors, `availability`, operation exit codes — and for each: what it actually measures, versus what a reader assumes it measures. My expectation is that the gap is documentable in a table, and that the table is the design doc.
2. **A decision on the default.** Whether `unknown` is a state these surfaces can express at all today. If it isn't, that's the smallest change with the widest reach, and everything else follows from it.
Happy to be wrong about this being one gap rather than eight — but the four proxy patterns recur across replication, components, backup, and CLI, which are otherwise unrelated code paths, and that's the part that reads as systemic rather than coincidental.
_Filed as a tracker at Kris's request while triaging the backlog. The instance issues keep their own priorities and milestones; nothing here blocks them._
Contributor guide
Research direction
Start by inventorying the named health-bearing surfaces: cluster_status, get_status, component health, clone/sync monitors, availability, and operation exit codes. Read the linked instance issues and compare what each surface measures with what it claims; done means producing the requested comparison table and deciding whether unknown is expressible today.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, cli, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100