HarperFast / HarperFast/harper
LMDB: TTL eviction of a record with a Blob attribute orphans the blob file on disk (unbounded disk growth; RocksDB reclaims correctly)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
On the **LMDB** storage engine, when a record with a `Blob` attribute expires via TTL, LMDB removes the row but leaves the blob **file** on disk. The orphaned files are never reclaimed → **unbounded disk growth** for any LMDB table combining TTL (`expiration`) with blob attributes. **RocksDB reclaims correctly** — this is engine-divergent.
## Reproduction (harper main `228eacc0f`, v5.1.15)
- Table `@table(expiration: 5)` with a `Blob` attribute, `threads: { count: 4 }`.
- Insert 5 records, each with a distinct ~200 KB blob.
- Confirm eviction: point-read → 404 (~4 s after the deadline) **before** counting.
- Count blob files under `{dataRootDir}/blobs/data` before vs after eviction.
| Engine | before → after |
|---|---|
| RocksDB | 5 → 0 (clean); replace-then-expire 1 → 0 (both v1 + v2 reclaimed) |
| **LMDB** | 5 → **5** (leak) |
Re-ran the LMDB case with a 30 s settle → 5 → **1**, not 0 — so this is **not** eviction lag; LMDB's unlink is partial/racy and permanently orphans blob files.
## Root-cause hypothesis (unverified, medium confidence)
`Table.evict() → removeEntry()` gates blob deletion on `entry.value && (metadataFlags & HAS_BLOBS)`, but the LMDB eviction path's `getEntry(id)` doesn't reliably return a populated value/flags, so `deleteBlobsInObject` never fires. Candidate fix: feed the scan's already-decoded record into `evict()` rather than re-fetching it.
## Severity
MEDIUM. LMDB is the deprecated engine (replication is RocksDB-only), so blast radius is limited to LMDB deployments — but it is a silent, unbounded-disk-growth footgun for anyone on LMDB with blob + TTL tables (sessions, caches, media with expiry).
Found via exploratory QA; local repro test available (`qa-scratch/qa-blob-ttl.test.ts`).
— KrAIs 🤖 (exploratory QA, on Kris's behalf)
Contributor guide
Research direction
Start with the local reproduction in qa-scratch/qa-blob-ttl.test.ts, then trace Table.evict() through removeEntry(), getEntry(id), and deleteBlobsInObject. Verify the LMDB TTL path reclaims blob files after eviction, including the reported replacement case, rather than leaving files under {dataRootDir}/blobs/data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100