HarperFast / HarperFast/harper
A transaction holding staged writes stays open for hours and is never reaped or identified (the holder behind the #2450 wedge)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Behind the #2450 wedge is a second defect that rocksdb-js 2.8.0 (#2466) does not touch: **a transaction with staged writes stayed open for hours on a production CM node, holding a verification-table (VT) write intent that nothing reaped, logged, or identified.** On 5.2.7 that held intent parked other writers indefinitely (the wedge). On 5.2.8 the park is bounded (rocksdb-js#744), so the same holder now costs a ~5 s stall on every colliding write and, if it also pins a RocksDB snapshot, blocks version reclamation until restart (the #2107 shape) — and it is still invisible.
Field evidence, 2026-09-01 and 2026-09-02, two nodes, 5.2.6/5.2.7 (details on #2450): commits parked for 4.5 h and 9 h; disk and CPU idle; no RocksDB `LOG` activity at either onset; replication-applied writes into the same database stalled at the same instant (database-wide, so the holder's slot collided with keys of several tables); **not one `Transaction was open too long and has been aborted` line on either node all day**, so the long-transaction monitor never reaped it.
## Why nothing reaps or names it today
- `resources/DatabaseTransaction.ts` (monitor): a write-bearing transaction past the 30 s idle limit is aborted **except when `sourceApply` or `isReplay`** — those keep the prior force-commit behavior and are never aborted, so a replication-apply transaction stuck mid-apply holds its intents for as long as it is stuck.
- rocksdb-js: a park (2.7.1) or a park timeout (2.8.0) records nothing about the holder. rocksdb-js#744 explicitly declined a `transaction:parkTimeout` event, so on 5.2.8 a permanently held intent surfaces only as the initiating write failing after `MAX_RETRIES × 5 s` with the generic coordinated-retries error.
- `checkOverloaded()` logs the *victim* (thread, table, `startedFrom`) once per stuck commit — never the transaction it is parked behind.
- rocksdb-js#768 (GC-release of dropped transactions) and #780 (reap at worker-env exit) cover two leak shapes; a holder that is **alive and referenced but stuck** (awaiting I/O, a blob stream, a lock) is covered by neither.
## Reproduction
The mechanism is deterministic at the rocksdb-js layer — script and output on [#2450 (comment)](https://github.com/HarperFast/harper/issues/2450#issuecomment-5510856864): a transaction that `putSync`s and never commits/aborts holds the slot; a `coordinatedRetry` commit that hits IsBusy parks on it forever on 2.7.1, wakes after 5005 ms on 2.8.0, and its retry then commits while the holder still holds. Scenario 1 (`holder alive, never commits`) *is* this issue: nothing in either version reports that holder. What is not reproduced is the production trigger — the transaction that stayed open for hours is unidentified, which is the point of the asks below.
## Asks
1. **Name the holder.** When a coordinated-retry park times out (rocksdb-js) or `checkOverloaded()` fires (core), log the holder's transaction id, `startedFrom` (resource + method), thread, and age. The `LockTracker` already knows its holders; the missing piece is surfacing the owning transaction.
2. **Report long-lived handles.** 2.8.0's `registryStatus().transactionDetails` exposes `id` + `ageMs` per live native handle; a periodic core warning for handles older than N minutes (with `startedFrom` when the wrapper is a `DatabaseTransaction`) would have identified this holder within minutes instead of never.
3. **Decide the reap policy for `sourceApply` / `isReplay` holders.** Today they are exempt for good reason (no resume path); at minimum they should be *reported* when they exceed the limit, and a stuck blob wait should release write intents (`abandonWrites()`) rather than hold them for the 900 s idle timeout.
Fix on `main` (v5.3) and backport to v5.2: the incident is on 5.2.x in production, and 5.2.8 leaves the holder undetectable.
Refs #2450, #2466, rocksdb-js#744, rocksdb-js#768, rocksdb-js#780, #2107, #2001.
Contributor guide
Assessment
This issue has not been assessed yet.