HarperFast / HarperFast/harper
RocksDB: audit history committed before an abrupt kill reads as zero on the next restart, then reappears
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
On RocksDB, audit history committed shortly before an **abrupt process kill** reads as **zero rows** on the immediately-following restart, then **reappears intact** after a later restart. The data is on disk the whole time; the first boot serves HTTP 200 with an empty audit history and no error, no warning, and nothing in the logs to distinguish it from a table that genuinely has no audit rows.
## Scope — deliberately narrowed
An earlier framing of this said audit history "may not survive a restart". A control arm refuted that broader claim and is worth stating up front so the fix is not aimed at the wrong path:
- **Clean restart is fine.** Identical fixture, 9,300 audited records, explicit flush, clean `killHarper` + `startHarper` on the same `dataRootDir`, no purge anywhere: `read_audit_log` and an independent server-side raw `getHistory` scan both read 9,300 before and 9,300 after, in **6/6 runs**.
- **Not purge-related.** A no-purge control reproduces the zero-read identically.
- **LMDB is clean** in every arm.
- An explicit flush after the writes, **or** a clean SIGTERM shutdown, each prevent it.
So the trigger is the abrupt-kill path specifically, not restart in general. That distinction matters because the two readings imply different fixes — a commit/flush durability gap versus a boot-time replay/visibility gap — and the transient nature (it comes back on the *next* restart) points at the latter.
## Why this is not covered by the existing fix
#1949 ("Crash recovery restores records but not the transaction log's committed watermark") closed 2026-08-24 via rocksdb-js PR #723 (`txnlog-committed-position-recovery`). That PR is an **ancestor of the rocksdb-js `v2.8.0` tag**, and the symptom above was still measured on 2.8.0 on 2026-09-01. So #1949's closing fix does not cover this.
This is also **not** #2337: that issue is about a *purged* `.txnlog` staying mapped in the live process. Here the log is unpurged and the first boot cannot see it.
## Mechanism
Not traced. That is the open work.
## What is not established
- Not bisected — no first-bad version.
- Single-node, default RocksDB engine, no cluster or replication arm.
- The count of restarts needed for the data to reappear was observed as "the next one"; whether a longer sequence ever loses it permanently was not tested.
Dup-searched for "audit history empty after restart", "committed watermark", and "recovery replay audit zero" across harper and harper-pro, open and closed: nothing matches this symptom.
*Found by exploratory QA (findings QA-793 and qa-wave-2026072609:1, merged — same defect reported twice); triaged 2026-09-02.*
Contributor guide
Research direction
Reproduce the abrupt-kill and restart sequence with the 9,300-record fixture, then trace the RocksDB recovery or replay path that determines audit-history visibility on the first boot. Done means the audit rows are visible immediately after restart and a regression test distinguishes this path from clean shutdown, purge, and LMDB controls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100