HarperFast / HarperFast/harper
Disk-space safety thresholds: reclamation / write-stop / hard-shutdown
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
Add a configurable disk-space safety model so Harper can deliberately throttle and then halt writes before disk exhaustion, instead of failing catastrophically when the disk fills (which is exceptionally difficult and time-consuming to recover from).
## Model (per Jeff Darnton's writeup)
Conceptually three thresholds:
1. **Reclamation threshold (X% free)** — reclamation runs on an interval; gets progressively more aggressive as disk pressure rises.
2. **Backpressure / warning** — clear warnings (logs at minimum) before unsafe levels are reached; "user" operations start receiving backpressure.
3. **Hard refusal threshold (Y% free)** — Harper refuses incoming **user** writes to protect itself. Internal/system writes (status flips, log rotation, replication acknowledgement, etc.) continue.
4. **Nuclear option** — if the disk would otherwise fill anyway, Harper shuts itself down before the filesystem hits 100% (recovery is far easier from a clean shutdown than from a full disk).
## Important constraint (per Kris)
Blocking *all* writes — including the `hdb_status` system table — would prevent us from setting the instance to unavailable. The throttle MUST be partitioned: "user" writes off, "system" writes on. Open question: does `hdb_analytics` count as system or user? (It's by far the most heavily written system table; consider moving it to its own database).
## Asks
- Config knobs: `storage.reclamationThreshold`, `storage.writeStopThreshold`, `storage.hardShutdownThreshold` (or equivalent).
- Apply the threshold against quota when one is configured (see [CORE-3011](https://harperdb.atlassian.net/browse/CORE-3011) / [HarperFast/harper#593](https://github.com/HarperFast/harper/issues/593)), not against raw filesystem.
- Partition writes: system writes always allowed; user writes throttled then refused.
- Auto-flip `hdb_status` to `unavailable` when the write-stop threshold trips.
- Clear logging of the state transitions.
## Acceptance criteria
- Operator can configure all three thresholds (or sensible defaults are documented).
- Reaching the write-stop threshold:
- User writes return a clear, actionable error.
- System writes (including `hdb_status` flip to unavailable) succeed.
- Logged at WARN or ERROR.
- Reaching the hard-shutdown threshold:
- Harper shuts down cleanly before disk hits 100%.
- Recovery path: operator clears space, restarts, instance comes back.
## Related
- [CORE-3011](https://harperdb.atlassian.net/browse/CORE-3011) / [HarperFast/harper#593](https://github.com/HarperFast/harper/issues/593) — use disk-space quota when computing free space (prerequisite for these thresholds to be quota-aware).
- [CORE-3007](https://harperdb.atlassian.net/browse/CORE-3007) / [HarperFast/harper-pro#169](https://github.com/HarperFast/harper-pro/issues/169) — containment-mode umbrella; related operator-controllable distress states.
---
Tracked in Jira: [CORE-3006](https://harperdb.atlassian.net/browse/CORE-3006)
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Assessment
This issue has not been assessed yet.