HarperFast / HarperFast/harper

Cross-worker write can race RocksDB table drop and poison catalog cleanup

Open
#1,381 3 comments 0 reactions 2 assignees Claimed by @kriszyp View on GitHub
area:storage bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary
Intermittently, a block of resources unit tests fail in their `before all` / `after all` hooks because `resetDatabases()` (called from `setupTestDBPath`) hits a column-family error while completing an interrupted table drop left over from a prior test. One run produced ~12 cascading hook failures from this single root cause. Re-running the job goes green (verified: attempt 2 passed on the same commit), so it's a test-isolation / state-leak flake.

## Exact errors
```
[main/0] [error]: Error committing cache update [Error: Transaction commit failed: Invalid argument: Invalid column family specified in write batch] { code: 'ERR_INVALID_ARGUMENT' }
[main/0] [error] [storage]: Failed to complete interrupted drop of table .TestTableForContext; will retry on next start Error: Remove failed: Invalid argument: Invalid column family specified in write batch
```
Stack:
```
Store.removeSync (@harperfast/rocksdb-js/src/store.ts:824)
RocksDatabase.removeSync (@harperfast/rocksdb-js/src/dbi.ts:461)
completeInterruptedDrop (resources/databases.ts:1638)
initStores (resources/databases.ts:541)
readRocksMetaDb (resources/databases.ts:439)
getDatabases (resources/databases.ts:331)
resetDatabases (resources/databases.ts:747)
setupTestDBPath (unitTests/testUtils.js:386)
```
Resulting cascade (sample): `"after all" hook for "should handle source.get returning null"`, `"before all" hook for "It can search_by_conditions on a resource"`, plus indexing/subscription/txn `before all` hooks.

## Why it's a flake
A prior test leaves a `TestTableForContext` drop half-completed (tombstoned) in the rocks meta DB; the next test's `setupTestDBPath → resetDatabases → readRocksMetaDb → initStores → completeInterruptedDrop` tries to finish it and the column family is already gone/inconsistent, so `removeSync` throws. Order/timing-dependent; green on re-run, same commit.

## Affected job / runtime
Observed on `Unit Test (Node.js v26)`, but the mechanism (shared-process unit run, leaked drop state) is not runtime-specific.

## Evidence
- PR #1363 run 27761092922 / job 82135389024 (attempt 1; attempt 2 green)

## Related
- #1276 (residual table-drop partial-failure risks — `completeInterruptedDrop` removing catalog rows on persistent CF-drop failure): same code path, but that's production-hardening; this is the test-isolation manifestation.
- #843 (runIndexing wedged after `clear()` on dropped CF): sibling "CF dropped out from under us" symptom.

---
_Filed by Claude (Opus 4.8) during CI flake triage while shepherding #1363/#1371/#1374._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.