RocksDB WriteBufferManager stall (writeBufferManagerAllowStall=true default) wedges every writer until restart — root cause of #2450

## Root cause of #2450: RocksDB WriteBufferManager write stall (`writeBufferManagerAllowStall: true` default) wedges every writer indefinitely

Harper defaults the process-wide RocksDB `WriteBufferManager` **on** at 1/3 of the block cache with **`allow_stall = true`** (`utility/rocksMemoryConfig.ts`, since 0183089 "Default the RocksDB WriteBufferManager on at 1/3 of block cache", 2026-06-11 — present in 5.2.5 through 5.2.8). When the sum of memtable memory across every database and column family in the process reaches that budget, RocksDB blocks **every** writer inside `DBImpl::WriteBufferManagerStallWrites()` until memory falls below the budget. On the production nodes it never does: the budget (~660 MiB on an unconstrained 8 GiB host) is spread over 28 column families whose individual memtables are each far below their 16 MiB flush trigger, so no column family flushes on its own, and RocksDB scheduled **no WBM-reason flush at all** (`rocksdb.flush.reason.write_buffer_manager COUNT : 0` in the LOG stats). Every writer sleeps on the stall condition variable forever; the `checkOverloaded()` guard observes a commit outstanding for 45 s and 503s the thread; the node stays wedged until restart. Both CM nodes hit this daily on 5.2.6, 5.2.7 and 5.2.8 — rocksdb-js 2.8.0 (#2466) could not fix it because the stall is inside RocksDB, below rocksdb-js.

### Evidence: gdb thread dump of a wedged production node (5.2.8, 2026-09-03 17:14 and 17:30 UTC, identical)

Main thread (a synchronous `putSync`):
```
#3 pthread_cond_wait
#4 rocksdb::port::CondVar::Wait()
#5 rocksdb::DBImpl::WriteBufferManagerStallWrites()
#6 rocksdb::DBImpl::PreprocessWrite(...)
#7 rocksdb::DBImpl::WriteImpl(...)
#8 rocksdb::DBImpl::Write(...)
#9 rocksdb::DB::Put(...)
#12 rocksdb_js::Database::PutSync(napi_env__*, napi_callback_info__*)
```
Two of the three per-database `rocksdb-commit` lanes:
```
#4 rocksdb::port::CondVar::Wait()
#5 rocksdb::DBImpl::WriteBufferManagerStallWrites()
#6 rocksdb::DBImpl::PreprocessWrite(...)
#7 rocksdb::DBImpl::WriteImpl(...)
#9 rocksdb::OptimisticTransaction::CommitWithParallelValidate()
#10 rocksdb::OptimisticTransaction::Commit()
#11 rocksdb_js::executeCommitWork(rocksdb_js::TransactionCommitState*)
```
The third lane: `rocksdb::WriteThread::LinkOne` → `CondVar::Wait()` (queued behind the stalled leader). http workers in `epoll` (alive — which is why reads kept working), RocksDB flush pool idle, RocksDB timer thread ticking (why the periodic stats dumps continued). Between the two captures the second lane went from idle to stalled: each database wedges at its next write, which is the staggered per-database onset seen on every incident. `rocksdb.stall.micros` stays 0 throughout because a WBM stall is not counted there.

Corroborating field facts from three days of incidents: `keys.written` frozen at onset on every affected database, CPU ~1 %, disk idle, zero blocked tasks, RocksDB `LOG` silent, the main thread frozen (stdout mirroring to `docker logs` stops at onset), and no `storage.rocks.*` overrides anywhere in the CM config.

### Reproduction (deterministic, ~2 s)

`rocksdb-js 2.8.0`, Node 24: configure a small WBM with `allowStall: true`, open 40 column families across 3 databases, `putSync` 8 KiB values round-robin. The process blocks inside one `putSync` and never returns; `lldb` shows the identical frame (`WBMStallInterface::Block` → `WriteBufferManagerStallWrites` → `PreprocessWrite` → `WriteImpl` → `Put` → `rocksdb_js::Database::PutSync`). The same run with `allowStall: false` completes. Script: `wbm-stall.mjs` (attached below). The small budget only compresses the time to reach the threshold; production reaches its ~660 MiB budget after hours of accumulation, which is why the wedge takes 4–12 h of uptime to appear.

wbm-stall.mjs

```js
// Reproduces HarperFast/harper#2450: RocksDB WriteBufferManager write stall (allowStall=true, Harper default).
// Many column families each hold a small memtable; their sum exceeds the shared WBM budget, but none is near
// its own 16 MiB flush trigger — so with allow_stall=true RocksDB stalls every writer in
// WriteBufferManagerStallWrites() and schedules no WBM-reason flush (prod: flush.reason.write_buffer_manager=0).
// putSync blocks the JS thread the same way the prod main thread (PutSync) and lanes (Commit) block.
// node wbm-stall.mjs [allowStall=1] [wbmBytes=2097152] [cfs=40]
import { RocksDatabase, versions } from '@harperfast/rocksdb-js';
import { mkdtempSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path';
const allowStall = (process.argv[2] ?? '1') !== '0';
const wbm = Number(process.argv[3] ?? 2 * 1024 * 1024);
const NCF = Number(process.argv[4] ?? 40);
RocksDatabase.config({ writeBufferManagerSize: wbm, writeBufferManagerAllowStall: allowStall, writeBufferManagerCostToCache: true });
console.log(`rocksdb-js ${versions['rocksdb-js']} (RocksDB ${versions.rocksdb}) | WBM=${wbm}B allowStall=${allowStall} CFs=${NCF}`);
const dir = mkdtempSync(join(tmpdir(), 'wbm-'));
const dbs = Array.from({ length: NCF }, (_, i) => RocksDatabase.open(join(dir, `db${i % 3}`), { name: `cf${i}` }));
const val = Buffer.alloc(8 * 1024, 1); // small values: fill many CFs a little, none near 16 MiB
const started = Date.now(); let n = 0, worst = 0;
for (; n < 500_000 && Date.now() - started < 40_000; n++) {
const db = dbs[n % dbs.length];
const t0 = Date.now(); db.putSync(Buffer.from(`k-${n}`), val); const dt = Date.now() - t0;
if (dt > worst) worst = dt;
if (dt > 8000) { console.log(`*** putSync #${n} BLOCKED ${dt}ms in the write path — WriteBufferManagerStallWrites(). ***`); console.log('RESULT: STALLED (reproduces #2450)'); process.exit(42); }
if (n % 5000 === 0) console.log(` ${n} putSync, ${Math.round((Date.now()-started)/1000)}s, worst ${worst}ms`);
}
console.log(`completed ${n} putSync, worst ${worst}ms — RESULT: ${worst > 8000 ? 'STALLED' : 'no stall'}`);
```

### Fix

1. **Default `writeBufferManagerAllowStall` to `false`** (rocksdb-js's own default; with it off, RocksDB schedules flushes more aggressively when the budget is exceeded instead of blocking writers — the budget becomes a soft cap, never a stall). This is the one-line fix for the outage.
2. If a hard cap is wanted, a stall must be paired with a guaranteed WBM-reason flush and a bound; RocksDB's `allow_stall` alone does not provide that when memory is spread across many column families (Kris: many CFs are "the main trigger for this issue").
3. Document the operator escape hatch: `storage.rocks.writeBufferManagerAllowStall: false` (or `writeBufferManagerSize: 0`).

Backport to v5.2: live production incidents on 5.2.6/5.2.7/5.2.8 (two CM nodes, daily). Supersedes the mechanism theory in #2450's earlier comments and makes #2471 moot as framed (there was no long-lived transaction holder; the "holder" was RocksDB's stall).

Refs #2450, #2471, #2466, #2001, #2321.