HarperFast / HarperFast/rocksdb-js

WBM stall safeguard (#755) is undone on reopen: TransactionDB::Open re-derives max_write_buffer_size_to_maintain=0 to 256 MiB per CF, permanent stall on Harper prod

Open
#821 1 comment 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
bug
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

## Summary

#755 drops `max_write_buffer_size_to_maintain` to `0` under a stalling WriteBufferManager so retained memtable history cannot fill the budget. RocksDB undoes that for every column family passed to `TransactionDB::Open`: `TransactionDB::PrepareWrap` rewrites `max_write_buffer_size_to_maintain == 0` to `-1`, which derives `max_write_buffer_number × write_buffer_size` = **256 MiB per CF** (`utilities/transactions/pessimistic_transaction_db.cc:296-300` in v11.8.1; `db_descriptor.cpp:1326` opens through `TransactionDB::Open`). Only CFs created *after* open keep the 0, which is why `test/write-buffer-manager-stall.test.ts` passes on a fresh DB. Every process restart reopens all CFs, so in production the safeguard is never in effect, and the permanent stall #755 was written to prevent is what took Harper's Central Manager down repeatedly on 2.8.0 (HarperFast/harper#2490, HarperFast/harper#2450).

## Production evidence (Harper 5.2.8 = rocksdb-js 2.8.0, RocksDB 11.8.1; 28 CFs across 3 DBs; WBM budget 661 MiB, `allowStall: true`)

Every 5.2.8 run logs `Options.max_write_buffer_size_to_maintain: 268435456` for **all** CFs. Memtable memory charged to the cache climbs with write volume and stops exactly at the budget; nothing can bring it back down because the excess is retained history (trimmed only *down to* the 256 MiB target, `memtable_list.cc TrimHistory`), the WBM flush trigger looks only at mutable memory (`write_buffer_manager.h:101-113`), and the stall ends only when total memory drops below the budget (`write_buffer_manager.cc:141-146`):

```
2026/09/02-17:56:31 WriteBuffer(1, 256.00 KB)
2026/09/02-19:56:31 WriteBuffer(827, 206.75 MB)
2026/09/02-22:16:31 WriteBuffer(1636, 409.00 MB)
2026/09/03-00:26:32 WriteBuffer(2236, 559.00 MB)
2026/09/03-04:56:32 WriteBuffer(2645, 661.25 MB) <- budget reached
2026/09/03-05:15:06 Harper: "Rejecting writes on this thread: a commit has been outstanding for 45000ms"
2026/09/03-12:46:34 WriteBuffer(2649, 662.25 MB) <- flat until restart
```
`rocksdb.flush.reason.write_buffer_manager = 0`, `rocksdb.stall.micros = 0`, "write-buffer-manager-limit-stops: 0" — the WBM stall is invisible to all three. gdb at wedge time: every writer in `WBMStallInterface::Block ← DBImpl::WriteBufferManagerStallWrites ← PreprocessWrite ← WriteImpl ← Put ← rocksdb_js::Database::PutSync` (and the `rocksdb-commit` lanes in the same place via `Commit`), flush pool idle.

## Reproduction (exact shipped binary: `harperfast/harper-pro:5.2.8`, 128 MiB budget, 6 CFs, 8 KiB puts)

```
phase 1 (fresh DB, CFs created after open): max_write_buffer_size_to_maintain per CF = {"0":5,"268435456":1} -> no stall
phase 2 (REOPEN, allowStall=true): {"268435456":7} -> putSync never returns
phase 2 (REOPEN, allowStall=false): {"268435456":7} -> completes; 12 "Write Buffer Manager" flushes
```

wbm-reopen-stall.mjs

```js
// node wbm-reopen-stall.mjs [seconds]
import { RocksDatabase, versions } from '@harperfast/rocksdb-js';
import { readFileSync, mkdirSync } from 'node:fs'; import { join } from 'node:path';
const [dir, phase, stallArg, wbmArg, cfArg, secArg] = process.argv.slice(2);
const allowStall = stallArg !== '0'; const wbm = Number(wbmArg ?? 128) * 1048576; const NCF = Number(cfArg ?? 6); const SECONDS = Number(secArg ?? 60);
RocksDatabase.config({ writeBufferManagerSize: wbm, writeBufferManagerAllowStall: allowStall, writeBufferManagerCostToCache: true });
mkdirSync(dir, { recursive: true });
const dbs = Array.from({ length: NCF }, (_, i) => RocksDatabase.open(join(dir, 'db'), { name: `cf${i}` }));
const log = () => { try { return readFileSync(join(dir, 'db', 'LOG'), 'utf8'); } catch { return ''; } };
const perCf = () => { const out = {}; for (const m of log().matchAll(/Options for column family \[([^\]]+)\][\s\S]*?max_write_buffer_size_to_maintain: (-?\d+)/g)) out[m[1]] = Number(m[2]); return out; };
const summary = {}; for (const v of Object.values(perCf())) summary[v] = (summary[v] ?? 0) + 1;
console.log(`phase ${phase}: rocksdb-js ${versions['rocksdb-js']} WBM=${wbm / 1048576}MiB allowStall=${allowStall} CFs=${NCF} | max_write_buffer_size_to_maintain {value: cfCount} = ${JSON.stringify(summary)}`);
const val = Buffer.alloc(8 * 1024, 1); const started = Date.now(); let n = 0, worst = 0, written = 0;
const finish = (verdict) => { const reasons = {}; for (const m of log().matchAll(/flush_reason": "([A-Za-z ]+)/g)) reasons[m[1]] = (reasons[m[1]] ?? 0) + 1; console.log(` written=${(written / 1048576).toFixed(0)}MiB puts=${n} worst putSync=${worst}ms | flushes: ${JSON.stringify(reasons)} | RESULT: ${verdict}`); };
const target = (phase === '1' ? 2 : 40) * NCF * 1048576;
for (; Date.now() - started < SECONDS * 1000 && written < target; n++) {
const db = dbs[n % NCF]; const t0 = Date.now(); db.putSync(Buffer.from(`p${phase}-k-${n}`), val); const dt = Date.now() - t0; written += val.length; if (dt > worst) worst = dt;
if (dt > 8000) { finish(`STALLED (putSync #${n} blocked ${dt}ms)`); process.exit(42); }
}
finish('no stall'); for (const db of dbs) db.close(); process.exit(0);
```
Run: `node wbm-reopen-stall.mjs /tmp/d 1 1 128 6 30 && timeout 100 node wbm-reopen-stall.mjs /tmp/d 2 1 128 6 80` → phase 2 hits the timeout (the JS thread is inside `putSync`, so not even the 8 s detector runs).

## Suggested fix

1. `resolveMaxWriteBufferSizeToMaintain` must return a small **positive** value, never 0 — RocksDB treats 0 as "unset" for transaction DBs. `1` (or one `writeBufferSize`) keeps the intent of #755; better is budget-aware, e.g. `writeBufferManagerSize / max(1, columnFamilyCount)` clamped to `[writeBufferSize, derived]`, so Σ history floors can never exceed the budget.
2. Add a **reopen** case to `write-buffer-manager-stall.test.ts`: create CFs, close, reopen, assert the LOG/`getOptions` value per CF and that writes do not stall.
3. Consider a hard guard: refuse (or log loudly) when `allowStall` is on and `columnFamilyCount × max_write_buffer_size_to_maintain > writeBufferManagerSize`.

Harper is currently running with `storage.rocks.writeBufferManagerAllowStall: false` as a workaround; with the stall off, memtable memory grows past the budget toward Σ min(256 MiB, written) per CF (~3 GiB on the CM's hot CFs), so the sizing fix is still needed even with stall disabled.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.