HarperFast / HarperFast/harper
sourceApply coordinated-retry loop has no backoff — hot spin re-encodes every write per round
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Source-applied (replication-apply) transactions deliberately never drop on conflict — correct, there is no resume path for a dropped write. But the coordinated-retry branch recurses into `commit()` with **no `delay()` at all**, unlike the `ERR_BUSY` branch which backs off up to `MAX_RETRY_DELAY_MS` (1s). See [v5.2.1 `resources/DatabaseTransaction.ts#L658-L675`](https://github.com/HarperFast/harper/blob/v5.2.1/resources/DatabaseTransaction.ts#L658-L675) — identical on main.
Once `retries > 0`, every write in the transaction is fully re-saved per round (`#L496-L500`): entry reload + decode, full merge/index-diff/audit-construction/encode, fresh options object and promise chain per recursion. The long-transaction monitor explicitly exempts `sourceApply` (`#L994`), so nothing bounds or surfaces the spin.
## Impact
A sustained local writer on a hot replicated key can pin a worker thread in a zero-backoff allocation loop indefinitely (source-vs-source contention serializes via apply-loop backpressure; local-vs-source does not). While spun, the apply loop is wedged: all peer legs for that database pause holding their in-flight WS buffers (`replication_maxPayload`, default 100MB each), and the pause-stall watchdog eventually reconnects and redelivers the same conflict. Contributed to heap pressure in an internal 16-node cluster incident.
## Suggested fix
Add the same bounded backoff the `ERR_BUSY` branch already has to the coordinated-retry branch — one `delay(Math.min(this.retries * this.retries, MAX_RETRY_DELAY_MS))` before the recursion. Never-drop semantics are preserved; only the spin rate changes.
---
🤖 Investigated and filed by Claude (Fable) on Nathan's behalf
Contributor guide
Research direction
Start in resources/DatabaseTransaction.ts at the coordinated-retry branch around lines 658-675, then compare it with the ERR_BUSY backoff and inspect the retry work around lines 496-500. Done means coordinated retries use the same bounded delay while preserving never-drop behavior; also review the sourceApply exemption around line 994.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100