HarperFast / HarperFast/harper

Concurrent blob REPLACE on one key leaks orphaned blob files (inline unlink loses the race; only cleanup_orphan_blobs recovers them)

Open
#1,832 1 comment 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

### Summary

Under **concurrent blob REPLACE on a single key**, the inline (commit-time) blob-unlink path loses the race and strands superseded blob files on disk. They are never reclaimed by any inline/background path — only an explicit `cleanup_orphan_blobs` operation recovers them. That manual sweep is a mitigation; it should not be required. This is a disk leak proportional to write concurrency (worst on the busiest tables), with **zero operator-visible signal** (no counter, no log, no scheduled sweep).

### Environment

- harper `main` @ `3dbcf7b9e` (v5.2.0-alpha.5)
- **Both storage engines** (RocksDB and LMDB — same signature)
- `threads.count = 4`

### Repro

10 concurrent writers × 8 `put()` REPLACEs each (80 total, plain `put()`, **no `ifVersion`**) of a ~2MB-blob record on **one** key.

Observed on-disk blob-file count:
- Converges 18 → **8** in ~4s, then sits **flat at 8** for the remaining 36s. Expected floor: **2** (or 1 after deleting the live record).
- Deleting the live record frees only its own current blob (8 → 7), leaving the already-orphaned superseded versions stranded.
- `cleanup_orphan_blobs` then reclaims **all** of them to the true floor (RocksDB 9 → 1, LMDB 10 → 1) within ~10s — confirming nothing is *permanently* unreferenced, but that reclamation only ever happens on the manual op.

**Sequential control** (80 REPLACEs @ 200ms, same fixture) settles to baseline+1 on both engines → **concurrency is the trigger**, not the fixture or a threshold.

Reproduction test (from our QA campaign, ready to promote as a regression anchor):
```
npm run test:integration -- "integrationTests/qa-scratch/qa592-blob-concurrent-writers.test.ts"
# LMDB leg: prefix HARPER_STORAGE_ENGINE=lmdb
```

### Root cause (traced)

The inline blob GC runs at commit (`resources/blob.ts` — `startPreCommitBlobsForRecord` / `cleanupUnusedBlobs`, and the on-demand reclaimer at `resources/blob.ts:1870` `cleanupOrphans`). Under concurrent REPLACE on the same key, the commit-time unlink of the superseded blob loses the race — the new version's file is written but the prior version's file is not consistently unlinked — so orphans accrue. There is no background sweep (cf. #1288 / F-006 for the adjacent pre-save-then-cleanup-on-lost-conflict path; here the **record is correct** and only file GC is deferred/lost, so the observable is distinct: a real on-disk leak, not a wrong record).

### Impact

- Unbounded on-disk growth on hot-key blob tables under concurrent writes, invisible until an operator happens to run `cleanup_orphan_blobs`.
- No orphan-bytes/count metric, log line, or scheduled sweep to make it observable.

### Ask

1. Make the inline/commit-time blob unlink race-safe so concurrent REPLACEs on one key do not strand superseded blob files (the primary fix — the manual sweep should not be load-bearing).
2. Failing/alongside (1), expose orphan blob bytes/count as an observable metric and/or an opt-in periodic sweep so the condition is detectable without running the op.

_Found by the exploratory QA campaign (QA-592, axis B×I). Filed at Kris's direction._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.