HarperFast / HarperFast/harper

RocksDB: audit history committed before an abrupt kill reads as zero on the next restart, then reappears

Open
#2,482 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

On RocksDB, audit history committed shortly before an **abrupt process kill** reads as **zero rows** on the immediately-following restart, then **reappears intact** after a later restart. The data is on disk the whole time; the first boot serves HTTP 200 with an empty audit history and no error, no warning, and nothing in the logs to distinguish it from a table that genuinely has no audit rows.

## Scope — deliberately narrowed

An earlier framing of this said audit history "may not survive a restart". A control arm refuted that broader claim and is worth stating up front so the fix is not aimed at the wrong path:

- **Clean restart is fine.** Identical fixture, 9,300 audited records, explicit flush, clean `killHarper` + `startHarper` on the same `dataRootDir`, no purge anywhere: `read_audit_log` and an independent server-side raw `getHistory` scan both read 9,300 before and 9,300 after, in **6/6 runs**.
- **Not purge-related.** A no-purge control reproduces the zero-read identically.
- **LMDB is clean** in every arm.
- An explicit flush after the writes, **or** a clean SIGTERM shutdown, each prevent it.

So the trigger is the abrupt-kill path specifically, not restart in general. That distinction matters because the two readings imply different fixes — a commit/flush durability gap versus a boot-time replay/visibility gap — and the transient nature (it comes back on the *next* restart) points at the latter.

## Why this is not covered by the existing fix

#1949 ("Crash recovery restores records but not the transaction log's committed watermark") closed 2026-08-24 via rocksdb-js PR #723 (`txnlog-committed-position-recovery`). That PR is an **ancestor of the rocksdb-js `v2.8.0` tag**, and the symptom above was still measured on 2.8.0 on 2026-09-01. So #1949's closing fix does not cover this.

This is also **not** #2337: that issue is about a *purged* `.txnlog` staying mapped in the live process. Here the log is unpurged and the first boot cannot see it.

## Mechanism

Not traced. That is the open work.

## What is not established

- Not bisected — no first-bad version.
- Single-node, default RocksDB engine, no cluster or replication arm.
- The count of restarts needed for the data to reappear was observed as "the next one"; whether a longer sequence ever loses it permanently was not tested.

Dup-searched for "audit history empty after restart", "committed watermark", and "recovery replay audit zero" across harper and harper-pro, open and closed: nothing matches this symptom.

*Found by exploratory QA (findings QA-793 and qa-wave-2026072609:1, merged — same defect reported twice); triaged 2026-09-02.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the abrupt-kill and restart sequence with the 9,300-record fixture, then trace the RocksDB recovery or replay path that determines audit-history visibility on the first boot. Done means the audit rows are visible immediately after restart and a regression test distinguishes this path from clean shutdown, purge, and LMDB controls.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.