HarperFast / HarperFast/harper-pro
cluster_status reports a stream healthy after it has lost transaction-log entries — nothing consumes getCorruptFrameReports()
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
Harper core now records every corrupt transaction-log frame it reads past and exposes them as `getCorruptFrameReports()` ([harper#2087](https://github.com/HarperFast/harper/pull/2087), `resources/replayLogsGuards.ts`). Nothing consumes it, so the only operator-visible signal that a replication stream has lost entries is still a log line.
That is the gap [harper#2016](https://github.com/HarperFast/harper/issues/2016) and [harper#2063](https://github.com/HarperFast/harper/issues/2063) were filed on: in both incidents `cluster_status` reported `connected: true` / `replicates: true` for days while the stream delivered nothing past a torn frame. #2063 ran 11 days that way. Nothing distinguished it from healthy replication except manually diffing record counts across nodes.
## What's needed
Surface the reports in `cluster_status` (and any component/health payload where a lost-entries condition belongs), so a stream that has amputated acknowledged entries is not reported as healthy.
The core API returns, per break site:
- `log` (scoped `/`), `logId`, `position`
- `midLog` — intact entries followed the break, so entries were lost rather than merely truncated
- `unreadableBytes`
- `stoppedIteration` — the most recent encounter ended iteration (torn tail, or the per-iteration resync cap)
- `firstSeen` / `lastSeen` / `occurrences`
Plus `getEvictedCorruptFrameReportCount()` for sites dropped past the retention bound.
## Two things to work out
**The map is per-worker-isolate.** A node-wide signal has to aggregate across worker threads — the reader that hit the break is whichever thread happened to drain that log. There is an existing cross-thread status collector (`componentStatus.crossThread`) that may be the right vehicle, or core may want to grow an aggregating helper; deciding that is part of this work and may mean a small core PR alongside.
**What the field should assert.** `midLog: true` means entries were lost from that log — it does not by itself mean the stream is currently stalled, since core now resyncs past the break (harper#2087 + [rocksdb-js#750](https://github.com/HarperFast/rocksdb-js/pull/750)). `stoppedIteration: true` on a `midLog` break is the stronger condition: entries after it are unreachable until the log is repaired or ages out. Both are worth reporting, but they warrant different words.
## Related
- [harper#2016 — Mid-log corrupt transaction-log frame silently truncates replay and replication — acknowledged writes lost](https://github.com/HarperFast/harper/issues/2016) — suggested directions 1 and 4 are this issue's operator-facing half
- [harper#2063 — Corrupt-log containment in RocksTransactionLogStore.getRange is per-drain, so one bad entry head-of-line-blocks a replication stream forever](https://github.com/HarperFast/harper/issues/2063)
- [harper#2087 — fix(replay): resync past a mid-log corrupt transaction-log frame and surface it as data loss](https://github.com/HarperFast/harper/pull/2087) — provides the API
- [rocksdb-js#750 — fix(txnlog): resync past a mid-log corrupt frame instead of ending the log](https://github.com/HarperFast/rocksdb-js/pull/750) — the reader half
- [rocksdb-js#748 — A failed transaction-log append orphans its partial bytes, baking a mid-file framing break into the log](https://github.com/HarperFast/rocksdb-js/issues/748) — prevention, not recovery
---
🤖 Filed by Claude Opus 5 on behalf of @kriszyp
Contributor guide
Research direction
Start by tracing cluster_status and componentStatus.crossThread, then inspect resources/replayLogsGuards.ts and the getCorruptFrameReports() API. Determine how worker-isolate reports are aggregated and define separate operator-visible states for midLog and stoppedIteration; done means cluster_status or a health payload exposes the retained and evicted reports without calling a damaged stream healthy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100