HarperFast / HarperFast/harper

Blob GC: files not deleted on audit-log expiry when using RocksDB audit store

Open
#708 0 comments 0 reactions 0 assignees View on GitHub
area:storage bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

## Summary

When a Blob-typed record is deleted via SQL and `auditRetention` is configured, the blob file on disk is never cleaned up if the audit store is backed by RocksDB (the default).

## Root cause

In `dist/resources/auditStore.js`, `scheduleAuditCleanup()` branches on audit store type:

```js
if (auditStore instanceof RocksTransactionLogStore) {
auditStore.rootStore.purgeLogs({ before: ... });
return; // <-- returns without blob file cleanup
}
// LMDB path: iterates audit entries and calls removeAuditEntry() which
// invokes deleteCallbacks to remove blob files
```

The RocksDB path calls `rootStore.purgeLogs()` and returns immediately. `removeAuditEntry()` — which is responsible for invoking blob-file delete callbacks — is never called. Additionally, `RocksTransactionLogStore.remove()` is a no-op with a TODO comment:

```js
async remove() {
// TODO: this function can likely be removed once the call to purgeLogs()
// is added in `resources/Table.ts`
}
```

## Effect

Blob files accumulate on disk indefinitely after the DB record is deleted, until the schema or table is dropped (which uses a direct filesystem operation, not the GC path).

## Confirmed via

Integration test `Blob lifecycle > blob file removed from filesystem after auditRetention expires` in `integrationTests/apiTests/blob.test.mjs` (PR #648). The test waits up to 35 seconds but blob files never disappear. `drop_schema` and `drop_table` tests that use direct filesystem operations pass fine.

## Workaround

Filesystem cleanup works when the schema or table is explicitly dropped (`drop_schema`, `drop_table` with `drop_records: true` — though the latter also appears incomplete). The audit-log GC path is the only broken case.

Related: #38 (Support running on Bun)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.