HarperFast / HarperFast/harper-pro
5.0.x only (fixed in 5.1 — backport harper#1251/#1252): sustained read pressure on a freshly-converted RocksDB node under replication write load escalates to OOM node death
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
### Summary
> **Scope: 5.0.x only — the fix already exists in the 5.1 line (harper#1251 + harper#1252, in `5.1.0-beta.3`) and is bench-verified to absorb this exact trigger (see verification comment). Ask = backport both to 5.0.x (→ 5.0.33).**
On 5.0.x, **sustained read pressure against a freshly-converted (LMDB→RocksDB) node that is applying replication write load can escalate to node death**: RocksDB snapshot-too-old conflict failures escape as `uncaughtException UnhandledPromiseRejection`, sequence numbers freeze, and the node is OOM-killed minutes later. Isolated by A/B on the in-place upgrade bench: with `COUNT(*)` polling every 5s the last-flipped node died **2/2 runs**; with the identical workload and 60s polling it survived cleanly (2/2). The upgrade procedure itself is fine — this is about what concurrent readers can do to a converting node.
### Signature (hot loop until exit 137 / OOMKilled=true)
```
[http/2] [error]: uncaughtException UnhandledPromiseRejection: ... "Error: Transaction commit failed: Operation failed. Try again.: Transaction could not check for conflicts for operation at SequenceNumber 2536790 as the MemTable only contains changes newer than SequenceNumber 2536925. Increasing the value of the max_write_buffer_size_to_maintain option could reduce the frequency of this error."
[http/1] [error]: Transaction was open too long and has been committed, from table: orders/ was started from orders.search
```
- `After 40 retries` (the #348 fingerprint) never appears — this is the snapshot-too-old path, not the retry-cap path.
- The long-held read snapshots from the polling (`Transaction was open too long …`) precede death in both failing runs and are the discriminating variable per the A/B.
- **This is the class harper#1251 + harper#1252 address in 5.1.0-beta.3** (WriteBufferManager default widens the MemTable window; the apply loop backpressures instead of escaping into unhandled rejections). Neither is in the 5.0.x line.
### Bench evidence (in-place scenario, image digest-verified, 4× 4.7.33 → 5.0.32)
80k×6 tables (480k records/node), identity pinned, 50 writes/s sustained:
| run | catch-up polling | last-flipped node |
|---|---|---|
| 8 | `COUNT(*)` every 5s | died mid-catch-up (~4 min post-flip), OOMKilled |
| 9 | `COUNT(*)` every 5s | converged to exact parity, then died ~6 min post-flip (242 conflict-loop lines), OOMKilled |
| 10 | every 60s | clean — exact parity ×4, alive through extended settle |
| 11 | every 60s | clean (confirmation leg) |
Survivors stayed mutually exact in all runs — the blast radius is the node, not peer data. Conversion itself is flat ~41–56s at this scale and not implicated.
### Why this matters in production
Monitoring dashboards, health checkers, and parity-verification tooling do exactly what the harness did — repeated table-scanning reads against nodes mid-upgrade. A reader must not be able to escalate to node death. During migration windows this is at its most likely: freshly-converted store, heavy replication apply traffic, and operators *intentionally* polling counts to verify progress.
### Asks (either unblocks; first is preferred)
1. **Backport harper#1251 + harper#1252 to 5.0.x** (→ 5.0.33) — fixes the class at the root.
2. Interim hardening: catch the snapshot-too-old rejection on the apply path (parallel to the #1252 approach) so it degrades to retry/backpressure instead of process death.
Operational mitigation until then: throttle COUNT-style polling (≥30–60s cadence) against nodes in conversion/catch-up windows.
### Repro
Lab in-place scenario: `V5_IMAGE=harperfast/harper-pro:5.0.32 SEED=80000 DBS=data,mig2 TABLES=orders,users,events RATE=50 POLL_INTERVAL=5 CATCHUP_TIMEOUT=1800 IP_NODE_MEM=3g incidents/inplace-upgrade-2026-06-11/run.sh` (~50 min; `POLL_INTERVAL=60` for the clean control). Happy to run candidate builds.
Related: #348 (sibling path, fixed in beta.3 by the same PRs), harper#1251, harper#1252, #352 (blocks the 5.1 alternative for in-place), #299.
Contributor guide
Assessment
This issue has not been assessed yet.