HarperFast / HarperFast/harper

Reclamation handler fills the disk it protects: unbounded stack-trace logging when statfs on ROOTPATH fails every tick (39.5 GB in ~13h)

Open
#2,289 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Harper's disk-space reclamation handler can FILL the disk: statfs error in `runReclamationHandlers` logs a stack trace per tick, unbounded

Observed on this host 2026-08-24. An orphaned ephemeral Harper (started by a test harness, later orphaned to ppid 1) wrote **39.5 GB** to `/log/hdb.log` over ~13 hours and took the machine from comfortable to **3 GiB free**.

The log tail is the same stack repeated to exhaustion:

```
at async statfs (node:internal/fs/promises:1828:18)
at async defaultGetAvailableSpaceRatio (…/node_modules/.bun/harper@5.2…)
at async Timeout.runReclamationHandlers (…/node_modules/.bun/harper@5.2…)
```

`runReclamationHandlers` fires on a timer, calls `defaultGetAvailableSpaceRatio` → `statfs()`, and when that rejects, the rejection is logged in full and the timer simply fires again. In our case the instance's data directory had gone away underneath the process (scratch dir removed while an orphan kept running), so `statfs` failed every tick, forever.

### Why this is worth fixing upstream
The handler exists to *protect* against low disk. In this failure mode it becomes the thing that consumes the disk — a monitoring loop whose error path is unbounded output. It is also self-accelerating in the sense that the closer the disk gets to full, the more likely other components start erroring into the same log.

### Suggested shape
1. Rate-limit or dedupe repeated identical failures in the reclamation loop (log once, then a periodic summary with a count).
2. Treat a persistent `statfs` failure on the data path as fatal-ish: the instance cannot do its job if its own ROOTPATH is gone — halt or self-disable the handler rather than logging forever.
3. Bound `hdb.log` (size-based rotation with a cap) so no single failure mode can consume a volume.

### Local context (not upstream's problem, but the reason we hit it)
The orphan existed because our ephemeral test lifecycle has a documented no-signal-handlers gap — the helper's own header records a prior 27 GB / four-day incident of the same family. Filing our side separately.

Reproduction sketch: start a Harper instance, delete its ROOTPATH out from under it, leave it running.

Contributor guide

Open the contributing guide

Research direction

Start at runReclamationHandlers and follow its call to defaultGetAvailableSpaceRatio and statfs when the ROOTPATH is removed. Reproduce the repeated failure by deleting the data directory under a running instance, then ensure persistent failures no longer produce unbounded hdb.log output and that the reclamation handler has a defined bounded failure behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.