HarperFast / HarperFast/harper
Cross-worker write can race RocksDB table drop and poison catalog cleanup
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Intermittently, a block of resources unit tests fail in their `before all` / `after all` hooks because `resetDatabases()` (called from `setupTestDBPath`) hits a column-family error while completing an interrupted table drop left over from a prior test. One run produced ~12 cascading hook failures from this single root cause. Re-running the job goes green (verified: attempt 2 passed on the same commit), so it's a test-isolation / state-leak flake.
## Exact errors
```
[main/0] [error]: Error committing cache update [Error: Transaction commit failed: Invalid argument: Invalid column family specified in write batch] { code: 'ERR_INVALID_ARGUMENT' }
[main/0] [error] [storage]: Failed to complete interrupted drop of table .TestTableForContext; will retry on next start Error: Remove failed: Invalid argument: Invalid column family specified in write batch
```
Stack:
```
Store.removeSync (@harperfast/rocksdb-js/src/store.ts:824)
RocksDatabase.removeSync (@harperfast/rocksdb-js/src/dbi.ts:461)
completeInterruptedDrop (resources/databases.ts:1638)
initStores (resources/databases.ts:541)
readRocksMetaDb (resources/databases.ts:439)
getDatabases (resources/databases.ts:331)
resetDatabases (resources/databases.ts:747)
setupTestDBPath (unitTests/testUtils.js:386)
```
Resulting cascade (sample): `"after all" hook for "should handle source.get returning null"`, `"before all" hook for "It can search_by_conditions on a resource"`, plus indexing/subscription/txn `before all` hooks.
## Why it's a flake
A prior test leaves a `TestTableForContext` drop half-completed (tombstoned) in the rocks meta DB; the next test's `setupTestDBPath → resetDatabases → readRocksMetaDb → initStores → completeInterruptedDrop` tries to finish it and the column family is already gone/inconsistent, so `removeSync` throws. Order/timing-dependent; green on re-run, same commit.
## Affected job / runtime
Observed on `Unit Test (Node.js v26)`, but the mechanism (shared-process unit run, leaked drop state) is not runtime-specific.
## Evidence
- PR #1363 run 27761092922 / job 82135389024 (attempt 1; attempt 2 green)
## Related
- #1276 (residual table-drop partial-failure risks — `completeInterruptedDrop` removing catalog rows on persistent CF-drop failure): same code path, but that's production-hardening; this is the test-isolation manifestation.
- #843 (runIndexing wedged after `clear()` on dropped CF): sibling "CF dropped out from under us" symptom.
---
_Filed by Claude (Opus 4.8) during CI flake triage while shepherding #1363/#1371/#1374._
Contributor guide
Assessment
This issue has not been assessed yet.