HarperFast / HarperFast/harper

Dropping a table leaves orphan blobs until the sweeper runs

Open
#595 1 comment 0 reactions 0 assignees View on GitHub
area:storage bug from-jira
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Dropping a table can leave orphan blobs on disk — blobs whose record references are no longer in any audit/active state but whose underlying files remain. Orphan-blob cleanup eventually removes them, but the gap between drop and cleanup is a leak.

## Reproduction (from Jira)

Cluster of two nodes (`ord01`, `ord02`):

1. Run a script that writes records containing blobs to `BlobHammerRun`.
2. Drop `BlobHammerRun` on `ord01`.
3. Wait for audit log to expire (without restarting Harper).
4. `ord01` audit is null (table dropped); `ord02` audit shows 1238 entries.
5. After audit expires: `ord01` audit null, `ord02` audit 2.
6. Restart Harper on both nodes.
7. `ord01` shows 528 orphan blobs (0 records, 2 audit entries); `ord02` shows 752 blobs with 1233 records and 2 audit entries.
8. Orphan-blob cleanup runs and deletes the 528 blobs from `ord01`.

Log evidence:
```
[http/2] [warn]: Checking for references to potential orphaned blobs in table BlobHammerRun
[http/2] [warn]: Checking for references to potential orphaned blobs in the audit log
[http/2] [warn]: Deleting 528 orphaned blobs
[http/2] [warn]: Finished deleting 528 orphaned blobs
[http/2] [warn]: Cleaned Orphan Blobs from BlobTest, deleted 528 blobs)
```

## Why this matters

The drop path should clean up the blobs it owns rather than leaving them for the orphan sweeper. Jira notes "we don't believe this has a production impact" — but it's the same root cause family as the broader [orphan blobs investigation](https://harperdb.atlassian.net/browse/CORE-3016), and the gap could matter on a node with many drops in flight or a slow orphan sweeper.

## Acceptance criteria

- Dropping a table eagerly deletes the blob files referenced only by records in that table (and only by audit entries that will not survive).
- The "wait for orphan sweep to find them" path remains as a safety net but is no longer the primary cleanup mechanism for drop-driven orphans.

## Related

- [CORE-3016](https://harperdb.atlassian.net/browse/CORE-3016) — Orphan blobs tracking & insights (lavinia's working notebook).

---

Tracked in Jira: [CORE-3003](https://harperdb.atlassian.net/browse/CORE-3003)

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.