HarperFast / HarperFast/harper

A failed audit-floor write blocks the boot purge exactly when the disk is full, making #1115's reclamation unreachable

Open
#2,486 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Split out of #2458 as a deliberate trade rather than a defect, so the PR does not have to settle it. Raised there by two review rounds (Chris Barber, and the graded leg of the cross-model round at `1f5ab5b`, which carried it forward as a minor).

## The mechanism

Every path that prunes audit history raises the retention floor **before** removing anything, and a throw from `raiseAuditFloor` is what stops the prune — that ordering is the whole guarantee of #2447, because a floor written *after* the removal is lost if the process dies in between, and the surviving lower floor then certifies a cursor whose history is gone.

`resources/auditStore.ts` → `purgeAgedLogs`:

```ts
if (isReadOnlyMode()) return [];
const before = Date.now() - auditRetention;
raiseAuditFloor((rootStore as any).auditStore, before); // throws → the purge below never runs
return rootStore.purgeLogs({ before });
```

## Why that is pointed here specifically

`purgeAgedLogs` is the one-shot purge wired into `replayLogs` for #1115: a node that crash-loops during startup replay never reaches the steady-state cleanup loop, so its aged backlog only grows and every recovery attempt gets more expensive. It exists to let such a node shed files *before* replaying.

The failure mode is therefore self-inflicted at the worst moment: **on a full volume, the thing that fails is the 8-byte floor write**, so the purge built to reclaim space becomes unreachable in precisely the condition it was built for. The node keeps crash-looping with a backlog it can no longer shed.

Not a boot crash — `resources/replayLogs.ts:79-86` already catches and warns, so recovery continues without the purge.

## Current call-site split (deliberate, and worth keeping in mind)

| Path | On a floor-write failure |
| --- | --- |
| `scheduleAuditCleanup` retention loop | swallowed and warned — availability wins; nothing is pruned, so the floor stays accurate |
| `purgeAgedLogs` (boot/recovery) | propagates; `replayLogs` catches and warns — purge skipped |
| `Table.deleteHistory` | propagates to the caller — correctness wins |
| `delete_transaction_logs_before` (`ResourceBridge`) | propagates to the caller — correctness wins |

## The constraint that rules out the obvious fix

"Just purge anyway and record that the floor is unknown" does not work, because **you cannot record that you pruned without recording if you cannot record.** The unknown sentinel is a write to the same store that just refused a write. Any escape has to avoid needing durable state at the moment of failure:

1. **Keep as-is.** Correctness-first; documented. Cost is the disk-full recovery path above.
2. **Reserve headroom for the marker** so the 8-byte write can always land — the floor record is fixed-size and there is exactly one per database, so this is bounded. Preserves the write-ahead invariant instead of trading it away.
3. **In-memory poison.** Let the reclamation paths purge, and mark that database's floor unknown *in the process* for its lifetime, so `oldestRetainedAuditTime()` fails closed until a floor write succeeds. Sheds the backlog without lying, but the honesty is not durable — a restart forgets it, and the persisted floor is then stale-low.
4. **Purge and accept an inaccurate floor.** Reintroduces the silent truncation #2447 exists to prevent. Listed for completeness; not recommended.

(2) looks the most promising, since it keeps the invariant rather than choosing which side of it to give up.

## Scope

Reclamation paths only — the boot purge, and arguably the retention loop. `deleteHistory` and the bridge operation should keep propagating: those are user-invoked, the caller can see the error, and there is no availability argument for pruning behind their back.

Reversible per call site, and no consumer reads the floor yet (#2448 is the first), so this can be settled without a migration.

Related: #2447 (the floor primitive), #2458 (the PR that introduced the ordering), #1115 (the boot purge this blocks), #2451, #2448.

Contributor guide

Open the contributing guide

Research direction

Start at resources/auditStore.ts, focusing on purgeAgedLogs, and compare its failure path with scheduleAuditCleanup, Table.deleteHistory, and delete_transaction_logs_before. Read resources/replayLogs.ts:79-86 and the related issues before evaluating the listed alternatives. Done means the selected reclamation behavior stays within scope, preserves the audit-floor invariant, and handles the full-volume boot-purge case.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.