HarperFast / HarperFast/harper
A failed audit-floor write blocks the boot purge exactly when the disk is full, making #1115's reclamation unreachable
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
Split out of #2458 as a deliberate trade rather than a defect, so the PR does not have to settle it. Raised there by two review rounds (Chris Barber, and the graded leg of the cross-model round at `1f5ab5b`, which carried it forward as a minor).
## The mechanism
Every path that prunes audit history raises the retention floor **before** removing anything, and a throw from `raiseAuditFloor` is what stops the prune — that ordering is the whole guarantee of #2447, because a floor written *after* the removal is lost if the process dies in between, and the surviving lower floor then certifies a cursor whose history is gone.
`resources/auditStore.ts` → `purgeAgedLogs`:
```ts
if (isReadOnlyMode()) return [];
const before = Date.now() - auditRetention;
raiseAuditFloor((rootStore as any).auditStore, before); // throws → the purge below never runs
return rootStore.purgeLogs({ before });
```
## Why that is pointed here specifically
`purgeAgedLogs` is the one-shot purge wired into `replayLogs` for #1115: a node that crash-loops during startup replay never reaches the steady-state cleanup loop, so its aged backlog only grows and every recovery attempt gets more expensive. It exists to let such a node shed files *before* replaying.
The failure mode is therefore self-inflicted at the worst moment: **on a full volume, the thing that fails is the 8-byte floor write**, so the purge built to reclaim space becomes unreachable in precisely the condition it was built for. The node keeps crash-looping with a backlog it can no longer shed.
Not a boot crash — `resources/replayLogs.ts:79-86` already catches and warns, so recovery continues without the purge.
## Current call-site split (deliberate, and worth keeping in mind)
| Path | On a floor-write failure |
| --- | --- |
| `scheduleAuditCleanup` retention loop | swallowed and warned — availability wins; nothing is pruned, so the floor stays accurate |
| `purgeAgedLogs` (boot/recovery) | propagates; `replayLogs` catches and warns — purge skipped |
| `Table.deleteHistory` | propagates to the caller — correctness wins |
| `delete_transaction_logs_before` (`ResourceBridge`) | propagates to the caller — correctness wins |
## The constraint that rules out the obvious fix
"Just purge anyway and record that the floor is unknown" does not work, because **you cannot record that you pruned without recording if you cannot record.** The unknown sentinel is a write to the same store that just refused a write. Any escape has to avoid needing durable state at the moment of failure:
1. **Keep as-is.** Correctness-first; documented. Cost is the disk-full recovery path above.
2. **Reserve headroom for the marker** so the 8-byte write can always land — the floor record is fixed-size and there is exactly one per database, so this is bounded. Preserves the write-ahead invariant instead of trading it away.
3. **In-memory poison.** Let the reclamation paths purge, and mark that database's floor unknown *in the process* for its lifetime, so `oldestRetainedAuditTime()` fails closed until a floor write succeeds. Sheds the backlog without lying, but the honesty is not durable — a restart forgets it, and the persisted floor is then stale-low.
4. **Purge and accept an inaccurate floor.** Reintroduces the silent truncation #2447 exists to prevent. Listed for completeness; not recommended.
(2) looks the most promising, since it keeps the invariant rather than choosing which side of it to give up.
## Scope
Reclamation paths only — the boot purge, and arguably the retention loop. `deleteHistory` and the bridge operation should keep propagating: those are user-invoked, the caller can see the error, and there is no availability argument for pruning behind their back.
Reversible per call site, and no consumer reads the floor yet (#2448 is the first), so this can be settled without a migration.
Related: #2447 (the floor primitive), #2458 (the PR that introduced the ordering), #1115 (the boot purge this blocks), #2451, #2448.
Contributor guide
Research direction
Start at resources/auditStore.ts, focusing on purgeAgedLogs, and compare its failure path with scheduleAuditCleanup, Table.deleteHistory, and delete_transaction_logs_before. Read resources/replayLogs.ts:79-86 and the related issues before evaluating the listed alternatives. Done means the selected reclamation behavior stays within scope, preserves the audit-floor invariant, and handles the full-volume boot-purge case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100