HarperFast / HarperFast/rocksdb-js
WBM stall safeguard (#755) is undone on reopen: TransactionDB::Open re-derives max_write_buffer_size_to_maintain=0 to 256 MiB per CF, permanent stall on Harper prod
- Dominant language
- C++
- Stars
- 21
- Forks
- 2
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 36
Description
## Summary
#755 drops `max_write_buffer_size_to_maintain` to `0` under a stalling WriteBufferManager so retained memtable history cannot fill the budget. RocksDB undoes that for every column family passed to `TransactionDB::Open`: `TransactionDB::PrepareWrap` rewrites `max_write_buffer_size_to_maintain == 0` to `-1`, which derives `max_write_buffer_number × write_buffer_size` = **256 MiB per CF** (`utilities/transactions/pessimistic_transaction_db.cc:296-300` in v11.8.1; `db_descriptor.cpp:1326` opens through `TransactionDB::Open`). Only CFs created *after* open keep the 0, which is why `test/write-buffer-manager-stall.test.ts` passes on a fresh DB. Every process restart reopens all CFs, so in production the safeguard is never in effect, and the permanent stall #755 was written to prevent is what took Harper's Central Manager down repeatedly on 2.8.0 (HarperFast/harper#2490, HarperFast/harper#2450).
## Production evidence (Harper 5.2.8 = rocksdb-js 2.8.0, RocksDB 11.8.1; 28 CFs across 3 DBs; WBM budget 661 MiB, `allowStall: true`)
Every 5.2.8 run logs `Options.max_write_buffer_size_to_maintain: 268435456` for **all** CFs. Memtable memory charged to the cache climbs with write volume and stops exactly at the budget; nothing can bring it back down because the excess is retained history (trimmed only *down to* the 256 MiB target, `memtable_list.cc TrimHistory`), the WBM flush trigger looks only at mutable memory (`write_buffer_manager.h:101-113`), and the stall ends only when total memory drops below the budget (`write_buffer_manager.cc:141-146`):
```
2026/09/02-17:56:31 WriteBuffer(1, 256.00 KB)
2026/09/02-19:56:31 WriteBuffer(827, 206.75 MB)
2026/09/02-22:16:31 WriteBuffer(1636, 409.00 MB)
2026/09/03-00:26:32 WriteBuffer(2236, 559.00 MB)
2026/09/03-04:56:32 WriteBuffer(2645, 661.25 MB) <- budget reached
2026/09/03-05:15:06 Harper: "Rejecting writes on this thread: a commit has been outstanding for 45000ms"
2026/09/03-12:46:34 WriteBuffer(2649, 662.25 MB) <- flat until restart
```
`rocksdb.flush.reason.write_buffer_manager = 0`, `rocksdb.stall.micros = 0`, "write-buffer-manager-limit-stops: 0" — the WBM stall is invisible to all three. gdb at wedge time: every writer in `WBMStallInterface::Block ← DBImpl::WriteBufferManagerStallWrites ← PreprocessWrite ← WriteImpl ← Put ← rocksdb_js::Database::PutSync` (and the `rocksdb-commit` lanes in the same place via `Commit`), flush pool idle.
## Reproduction (exact shipped binary: `harperfast/harper-pro:5.2.8`, 128 MiB budget, 6 CFs, 8 KiB puts)
```
phase 1 (fresh DB, CFs created after open): max_write_buffer_size_to_maintain per CF = {"0":5,"268435456":1} -> no stall
phase 2 (REOPEN, allowStall=true): {"268435456":7} -> putSync never returns
phase 2 (REOPEN, allowStall=false): {"268435456":7} -> completes; 12 "Write Buffer Manager" flushes
```
wbm-reopen-stall.mjs
```js
// node wbm-reopen-stall.mjs [seconds]
import { RocksDatabase, versions } from '@harperfast/rocksdb-js';
import { readFileSync, mkdirSync } from 'node:fs'; import { join } from 'node:path';
const [dir, phase, stallArg, wbmArg, cfArg, secArg] = process.argv.slice(2);
const allowStall = stallArg !== '0'; const wbm = Number(wbmArg ?? 128) * 1048576; const NCF = Number(cfArg ?? 6); const SECONDS = Number(secArg ?? 60);
RocksDatabase.config({ writeBufferManagerSize: wbm, writeBufferManagerAllowStall: allowStall, writeBufferManagerCostToCache: true });
mkdirSync(dir, { recursive: true });
const dbs = Array.from({ length: NCF }, (_, i) => RocksDatabase.open(join(dir, 'db'), { name: `cf${i}` }));
const log = () => { try { return readFileSync(join(dir, 'db', 'LOG'), 'utf8'); } catch { return ''; } };
const perCf = () => { const out = {}; for (const m of log().matchAll(/Options for column family \[([^\]]+)\][\s\S]*?max_write_buffer_size_to_maintain: (-?\d+)/g)) out[m[1]] = Number(m[2]); return out; };
const summary = {}; for (const v of Object.values(perCf())) summary[v] = (summary[v] ?? 0) + 1;
console.log(`phase ${phase}: rocksdb-js ${versions['rocksdb-js']} WBM=${wbm / 1048576}MiB allowStall=${allowStall} CFs=${NCF} | max_write_buffer_size_to_maintain {value: cfCount} = ${JSON.stringify(summary)}`);
const val = Buffer.alloc(8 * 1024, 1); const started = Date.now(); let n = 0, worst = 0, written = 0;
const finish = (verdict) => { const reasons = {}; for (const m of log().matchAll(/flush_reason": "([A-Za-z ]+)/g)) reasons[m[1]] = (reasons[m[1]] ?? 0) + 1; console.log(` written=${(written / 1048576).toFixed(0)}MiB puts=${n} worst putSync=${worst}ms | flushes: ${JSON.stringify(reasons)} | RESULT: ${verdict}`); };
const target = (phase === '1' ? 2 : 40) * NCF * 1048576;
for (; Date.now() - started < SECONDS * 1000 && written < target; n++) {
const db = dbs[n % NCF]; const t0 = Date.now(); db.putSync(Buffer.from(`p${phase}-k-${n}`), val); const dt = Date.now() - t0; written += val.length; if (dt > worst) worst = dt;
if (dt > 8000) { finish(`STALLED (putSync #${n} blocked ${dt}ms)`); process.exit(42); }
}
finish('no stall'); for (const db of dbs) db.close(); process.exit(0);
```
Run: `node wbm-reopen-stall.mjs /tmp/d 1 1 128 6 30 && timeout 100 node wbm-reopen-stall.mjs /tmp/d 2 1 128 6 80` → phase 2 hits the timeout (the JS thread is inside `putSync`, so not even the 8 s detector runs).
## Suggested fix
1. `resolveMaxWriteBufferSizeToMaintain` must return a small **positive** value, never 0 — RocksDB treats 0 as "unset" for transaction DBs. `1` (or one `writeBufferSize`) keeps the intent of #755; better is budget-aware, e.g. `writeBufferManagerSize / max(1, columnFamilyCount)` clamped to `[writeBufferSize, derived]`, so Σ history floors can never exceed the budget.
2. Add a **reopen** case to `write-buffer-manager-stall.test.ts`: create CFs, close, reopen, assert the LOG/`getOptions` value per CF and that writes do not stall.
3. Consider a hard guard: refuse (or log loudly) when `allowStall` is on and `columnFamilyCount × max_write_buffer_size_to_maintain > writeBufferManagerSize`.
Harper is currently running with `storage.rocks.writeBufferManagerAllowStall: false` as a workaround; with the stall off, memtable memory grows past the budget toward Σ min(256 MiB, written) per CF (~3 GiB on the CM's hot CFs), so the sizing fix is still needed even with stall disabled.
Contributor guide
Assessment
This issue has not been assessed yet.