HarperFast / HarperFast/harper

Investigate RocksDB space reclamation after bulk delete / TTL churn (bottommost tombstones not reclaimed by plain compaction)

Open
#1,384 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

While building a free-page-reclamation integration test (HarperFast/harper#1378, case S4), we observed that after deleting **all** rows from a RocksDB-backed table, a plain compaction does **not** reclaim bottommost-level (L6) SST space. RocksDB's default `bottommost_level_compaction = kIfHaveCompactionFilter` means that, without a registered compaction filter, tombstones at the bottommost level are left in place: ~19 MB of dead SST persisted after deleting a 100K-row table (1 primary + 5 secondary-index column families). Only an explicit `store.clear()` (`compactRange()` + `DeleteFilesInRange()`) removed the files.

## Why it matters

This is the RocksDB analog of the **LMDB free-page accumulation** seen across the fleet (e.g. 9.4 GB file / 23K live records under 1h TTL on the automotive-risk cluster; 6.1 GB / 576 MB live on the redirect cluster). The motivating pattern is exactly **delete-heavy / TTL-churn** workloads (the ST-1 stress gap). If freed space is neither reclaimed nor reused under sustained delete/expire churn, disk can grow unbounded.

## Open questions

1. Is this expected RocksDB behavior — freed space is **reused** by future writes rather than returned to the OS — and therefore fine for steady-state churn?
2. Under sustained TTL-eviction / delete-heavy load, does Harper's normal background compaction eventually reclaim or reuse bottommost space, or does the data dir grow unbounded?
3. Should Harper register a compaction filter (or periodic full compaction / `DeleteFilesInRange`) for delete/TTL-heavy tables so bottommost tombstones are reclaimed?
4. Timing nuance: Harper's `DELETE` on RocksDB writes a null-value "soft-delete marker" to the primary store; a background cleanup scan later calls `remove()` to create the actual RocksDB tombstone. The ordering of that scan vs. compaction affects when (if) space is reclaimed.

## Evidence

- HarperFast/harper#1378, case S4 (now `skip`ped pending this investigation) + its fixture probe.
- Observed (100K rows): ~19 MB live SST after `DELETE` + plain `compact()`; → 0 after explicit `clear()`.

## Next step

Decide the intended semantics (reuse vs. reclaim), then either (a) document it and make S4 a "space is reused, not unbounded" test (insert → delete → re-insert → assert no ~doubling), and/or (b) run the free-page reclaim test under the **LMDB** engine where the original fleet concern lives, and/or (c) add a compaction-filter / periodic-reclaim mechanism for delete/TTL-heavy tables.

Related: HarperFast/harper#1378 (S4), Release Testing Strategy §6.3 (ST-1), Harper v5 Integration Test Plan §5 (ST-1).

Filed by Claude (Opus 4.8) on behalf of Kris while reviewing the release-testing coverage gaps.

Contributor guide

Open the contributing guide

Research direction

Start with the HarperFast/harper#1378 S4 fixture probe and compare the observed results of plain compact() with explicit clear() after bulk deletion. Determine whether sustained delete/TTL churn reuses space or leaves it growing, then define the outcome as documented semantics and a passing reuse/reclamation test; the issue does not name a source file or test path.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.