HarperFast / HarperFast/harper
Investigate RocksDB space reclamation after bulk delete / TTL churn (bottommost tombstones not reclaimed by plain compaction)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
While building a free-page-reclamation integration test (HarperFast/harper#1378, case S4), we observed that after deleting **all** rows from a RocksDB-backed table, a plain compaction does **not** reclaim bottommost-level (L6) SST space. RocksDB's default `bottommost_level_compaction = kIfHaveCompactionFilter` means that, without a registered compaction filter, tombstones at the bottommost level are left in place: ~19 MB of dead SST persisted after deleting a 100K-row table (1 primary + 5 secondary-index column families). Only an explicit `store.clear()` (`compactRange()` + `DeleteFilesInRange()`) removed the files.
## Why it matters
This is the RocksDB analog of the **LMDB free-page accumulation** seen across the fleet (e.g. 9.4 GB file / 23K live records under 1h TTL on the automotive-risk cluster; 6.1 GB / 576 MB live on the redirect cluster). The motivating pattern is exactly **delete-heavy / TTL-churn** workloads (the ST-1 stress gap). If freed space is neither reclaimed nor reused under sustained delete/expire churn, disk can grow unbounded.
## Open questions
1. Is this expected RocksDB behavior — freed space is **reused** by future writes rather than returned to the OS — and therefore fine for steady-state churn?
2. Under sustained TTL-eviction / delete-heavy load, does Harper's normal background compaction eventually reclaim or reuse bottommost space, or does the data dir grow unbounded?
3. Should Harper register a compaction filter (or periodic full compaction / `DeleteFilesInRange`) for delete/TTL-heavy tables so bottommost tombstones are reclaimed?
4. Timing nuance: Harper's `DELETE` on RocksDB writes a null-value "soft-delete marker" to the primary store; a background cleanup scan later calls `remove()` to create the actual RocksDB tombstone. The ordering of that scan vs. compaction affects when (if) space is reclaimed.
## Evidence
- HarperFast/harper#1378, case S4 (now `skip`ped pending this investigation) + its fixture probe.
- Observed (100K rows): ~19 MB live SST after `DELETE` + plain `compact()`; → 0 after explicit `clear()`.
## Next step
Decide the intended semantics (reuse vs. reclaim), then either (a) document it and make S4 a "space is reused, not unbounded" test (insert → delete → re-insert → assert no ~doubling), and/or (b) run the free-page reclaim test under the **LMDB** engine where the original fleet concern lives, and/or (c) add a compaction-filter / periodic-reclaim mechanism for delete/TTL-heavy tables.
Related: HarperFast/harper#1378 (S4), Release Testing Strategy §6.3 (ST-1), Harper v5 Integration Test Plan §5 (ST-1).
Filed by Claude (Opus 4.8) on behalf of Kris while reviewing the release-testing coverage gaps.
Contributor guide
Research direction
Start with the HarperFast/harper#1378 S4 fixture probe and compare the observed results of plain compact() with explicit clear() after bulk deletion. Determine whether sustained delete/TTL churn reuses space or leaves it growing, then define the outcome as documented semantics and a passing reuse/reclamation test; the issue does not name a source file or test path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100