HarperFast / HarperFast/harper

cleanup_orphan_blobs always throws on zero-table databases but returns 200 "cleanup started" — recovery sweeper silently never runs

Open
#2,390 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## What happens

`cleanup_orphan_blobs` on a database whose tables have all been dropped (zero-table database) returns `200 "Orphaned blobs cleanup started"` and then **always throws internally** — the sweeper silently never runs for that database, and the operator's only signal is a success response. Any orphaned blob files in that database's blob roots are unreclaimable through the operations API, permanently.

## Mechanism (on `origin/main` @ e16d9616)

`resources/blob.ts` `cleanupOrphans()` (line 2792) resolves `store`/`auditStore` only inside `for (const tableName in database)` — an empty table map leaves `store` undefined, and `getRootBlobPathsForDB()` (line 2074) throws `'No store specified, can not determine blob storage path'`. The caller `dataLayer/schema.ts:370` (`cleanupOrphanBlobs`) fires-and-forgets the promise and unconditionally returns the success message.

Reproduced 3/3 from cold on `2615b092b`; the mechanism is unchanged on today's main. Draft PR #1835 (dry-run + orphan counts for #1832) touches `cleanupOrphans` extensively but leaves the store-resolution loop byte-for-byte unchanged — it adds a `.catch(logger.error)` at the call site, so post-#1835 the throw becomes a logged error, but zero-table databases still can never be swept.

## Why it matters

This is masked today only because `dropTable()`'s eager `deleteBlobsInObject()` usually does the work first. Every scenario where that misses — crash mid-drop (the ~100–400 ms unlink window, see #595), or the clustered drop shape — lands orphans in exactly the databases this bug makes unsweepable. The recovery path has a hole precisely where the crash scenarios need it. (harper-pro's `blob-drop-crash-orphan-window.test.ts` docstring independently cites this same empty-database gap as prior finding QA-697.)

## Suggested fix

Resolve the blob roots from the database handle before the table loop (the roots exist independently of live tables), or fail the operation loudly instead of returning "started".

## Existing work checked

#1832 (orphan observability umbrella, OPEN) + draft PR #1835 — confirmed not to fix this. #595 (drop_table orphans until sweeper runs) — the scenario this bug compounds. No issue tracks the zero-table throw ("zero table", "No store specified", "cleanup_orphan_blobs empty database" searches: no hit).

## What this does not prove

Whether blob roots shared across databases change the fix shape; LMDB blob-store variant untested.

---
*From dispatch QA finding F-196, verified against harper origin/main e16d9616, 2026-08-29.*

Contributor guide

Open the contributing guide

Research direction

Start in resources/blob.ts at cleanupOrphans() and follow its store-resolution loop, then inspect dataLayer/schema.ts at cleanupOrphanBlobs() for the fire-and-forget response. Reproduce the zero-table database case described in the issue; done means the recovery sweep no longer fails silently and the API outcome matches whether cleanup started.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.