HarperFast / HarperFast/rocksdb-js

Optimistic commit racing DropColumnFamily latches a fatal background error under the default parallel OCC validation

Open
#806 2 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
bug
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

## What

An **optimistic** transaction whose commit races a `DropColumnFamily` from another thread does not fail at conflict validation as [#725](https://github.com/HarperFast/rocksdb-js/pull/725) assumed. Under RocksDB's default `OccValidationPolicy::kValidateParallel` the commit is admitted past validation, fails inside the memtable inserter with `Invalid argument: Invalid column family specified in write batch`, and `DBImpl::HandleMemTableInsertFailure` latches that as a **fatal background error on the whole environment**. From that instant every write to every column family on the path fails with the same message until the database is closed and reopened (or, on 2.8.0, `resume()`d).

#725 fixed this for non-transactional `putSync`/`removeSync` (`ignore_missing_column_families`) and kept transactions out of scope on the premise that

> Optimistic mode (the default, and the only mode Harper uses) already fails safely: conflict validation rejects a commit naming a dropped family early, with `Invalid argument: Could not access column family `

That is only true of `kValidateSerial`. [#726](https://github.com/HarperFast/rocksdb-js/issues/726) tracks the same poisoning for pessimistic mode; this is the optimistic-mode counterpart, and it is the one Harper hits.

## Why

`OptimisticTransactionDB::Open(dbOptions, path, cfDescriptors, ...)` (`db_descriptor.cpp`, the overload without `OptimisticTransactionDBOptions`) leaves `validate_policy` at its default, `kValidateParallel`. `OptimisticTransaction::Commit()` then takes `CommitWithParallelValidate()`:

```cpp
Status s = TransactionUtil::CheckKeysForConflicts(db_impl, *tracked_locks_, true /* cache_only */);
if (!s.ok()) return s; // "Could not access column family N" only if already dropped HERE
s = db_impl->Write(write_options_, GetWriteBatch()->GetWriteBatch()); // plain Write, no callback
```

Validation runs **before** the commit enters the write thread. `DBImpl::DropColumnFamilyImpl` serializes only against write groups (`write_thread_.EnterUnbatched` around `LogAndApply`, which is where `cfd->SetDropped()` removes the family from the set), so a drop that starts after validation and holds the write thread through its MANIFEST write lands before the commit's `Write()` gets its turn. The batch still names the dropped family, `MemTableInserter::SeekToColumnFamily` fails it (the transaction's `WriteOptions` deliberately do not set `ignore_missing_column_families`), and `HandleMemTableInsertFailure` calls `SetBGError(status, BackgroundErrorReason::kMemTable)` — fatal severity, `IsDBStopped()` true, every later `PreprocessWrite` returns the stored status.

With `kValidateSerial` the validation is the write-group callback (`WriteWithCallback`), which runs inside the write thread, so the drop cannot interleave and the commit fails with the attributable `Could not access column family N` that #725 measured.

RocksDB's own LOG from a poisoned Harper run shows the two events 63 µs apart on two threads (main thread dropping, the commit lane committing):

```
[db/db_impl/db_impl.cc:3951] Dropped column family with id 3
[db/error_handler.cc:369] ErrorHandler: Set regular background error, auto_recovery=0, stop=1
```

## Reproducer (binding only, rocksdb-js 2.7.1, poisons on the first round)

```js
import { Worker, isMainThread, parentPort, workerData } from 'node:worker_threads';
import { RocksDatabase } from '@harperfast/rocksdb-js';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

if (isMainThread) {
const path = mkdtempSync(join(tmpdir(), 'cf-race-'));
const meta = RocksDatabase.open(path, { name: 'meta' });
const worker = new Worker(new URL(import.meta.url), { workerData: { path } });
worker.on('message', (m) => {
if (m.type === 'ready') {
const table = RocksDatabase.open(path, { name: 'table' });
worker.postMessage('commit');
setImmediate(() => table.dropSync()); // lands between the commit's validation and its Write()
} else {
console.log('commit error:', m.error);
try { meta.putSync('probe', 1); console.log('env healthy'); }
catch (e) { console.log('ENV POISONED ->', e.message); }
worker.terminate(); meta.close();
}
});
} else {
const table = RocksDatabase.open(workerData.path, { name: 'table' });
parentPort.once('message', async () => {
let error;
try { await table.transaction((t) => t.putSync('k', Buffer.alloc(4096, 1))); } catch (e) { error = e.message; }
parentPort.postMessage({ type: 'done', error });
});
parentPort.postMessage({ type: 'ready' });
}
```

```
commit error: Transaction commit failed: Invalid argument: Invalid column family specified in write batch
ENV POISONED -> Put failed: Invalid argument: Invalid column family specified in write batch
```

Note the commit's own error is the poison message, not `Could not access column family` — the write got past validation.

## Impact

Harper drops a table's column families from the thread that runs `drop_table` while other worker threads can still be committing to that table (a source-fill cache write that resolved its caller before committing, an eviction batch, a plain `put` admitted before the schema change reached that worker). One such commit turns every Harper database on that path read-only until restart: the drop's own catalog cleanup fails and leaves the `dropping` tombstone, every worker's interrupted-drop completion fails on the latched error, and a same-name create cannot proceed. Tracked on the Harper side as [harper#1381](https://github.com/HarperFast/harper/issues/1381) (public CI runs 33076221779 and 33139269321 on `main`, `integrationTests/apiTests/blob.test.mjs`). Affects 2.7.1 (what Harper `main` pins) and 2.8.0 — the open path is unchanged there; 2.8.0's `getLastError()`/`resume()` only adds observability and recovery after the fact.

## Fix directions

1. **Serialize drops against in-flight optimistic commits in the binding** — the interlock #726 describes (per-column-family in-flight touch count taken on transactional `Put`/`Delete`, released on commit/rollback; `Database::Drop`/`DropSync` wait for it to reach zero, coordinated with `columnsMutex` and the `CommitWorker` lane). Fixes both modes with no change to the commit path's parallelism.
2. **Open with `OptimisticTransactionDBOptions{ validate_policy = kValidateSerial }`** — a one-line change that moves validation into the write thread, restoring the behaviour #725 assumed. Costs commit parallelism under contention (validation joins the leader's critical section), which is why RocksDB made parallel the default.
3. `ignore_missing_column_families` on transaction write options is **not** an option: #725 measured it turning a commit that spans a live and a dropped family into a silent partial commit.

Either 1 or 2 makes a drop/commit race a contained, attributable failure of that one commit instead of a node-wide write outage.

---
_Filed by Claude Fable 5 while root-causing harper#1381._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.