HarperFast / HarperFast/rocksdb-js

WriteBufferManager stall is unobservable and unrecoverable: no counter moves, no log line, only a restart clears it

Open
#822 0 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

## Summary

A WriteBufferManager write stall is invisible to every counter and log surface this library exposes, and nothing ever ends one. On a wedged Harper Central Manager node (5.2.8 = rocksdb-js 2.8.0, 28 CFs, 661 MiB budget, `allowStall: true`) all three of the obvious signals read zero for the entire eight-hour wedge:

| Signal | Value during the stall |
|---|---|
| `rocksdb.stall.micros` | 0 — a WBM stall is not counted there |
| `rocksdb.flush.reason.write_buffer_manager` | 0 |
| `write-buffer-manager-limit-stops` (LOG) | 0 |

Meanwhile every writer thread sat in `WBMStallInterface::Block ← DBImpl::WriteBufferManagerStallWrites ← PreprocessWrite ← WriteImpl ← Put ← rocksdb_js::Database::PutSync`, the flush pool was idle, and reads kept working — so from outside the process the node looks healthy while no write can complete. Diagnosis required gdb on production. That is the part this issue is about; the sizing defect that filled the budget is #821 and the Harper default that enables stalling is HarperFast/harper#2490.

Two things are missing:

**1. No way to see it.** `WriteBufferManager` exposes `memory_usage()`, `mutable_memtable_memory_usage()`, `buffer_size()` and `IsStallActive()`, none of which are surfaced. A stall that has been active for minutes is indistinguishable from an idle database.

**2. No way out of it.** The stall ends only when total memtable memory drops below the budget, and `ShouldFlush()` requires `mutable >= buffer_size / 2` — so when the budget is held by retained history or by memtables already awaiting flush, no flush is triggered, and no stalled writer can switch a memtable to cause one. The only exit is a process restart.

## Requested

1. **Expose the manager's live state.** `memoryUsage`, `mutableMemoryUsage`, `bufferSize`, `stallActive` on the existing stats/settings surface, so a stall is visible in Harper's `system_information` rather than in a debugger.
2. **Watchdog log.** When a stall has been active longer than some threshold (a few seconds), log once at `warn`: budget, memory usage, mutable share, live CF count, and the per-CF `max_write_buffer_size_to_maintain` in effect. Every one of those was needed to explain the production wedge, and every one required reading the RocksDB `LOG` or attaching a debugger.
3. **Optional self-heal.** `WriteBufferManager::SetAllowStall(false)` calls `MaybeEndWriteStall()`, which releases the queued writers — so a bounded, opt-in escape hatch is available without a restart (drop stalling after N seconds, log loudly, and never re-arm silently). Worth discussing rather than assuming: it converts a hang into unbounded memtable growth, which is the right trade for a wedged production node but not obviously the right default.

Item 1 is useful regardless of how #821 lands. Items 2 and 3 are the "if this ever happens again, we should not need gdb" half.

## Not in scope

The sizing/retention defect (#821), the runtime-mutation and late-CF bypass paths (#781), and Harper's `writeBufferManagerAllowStall: true` default (HarperFast/harper#2490).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.