HarperFast / HarperFast/harper

sourceApply coordinated-retry loop has no backoff — hot spin re-encodes every write per round

Open Beginner friendly
#2,141 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Source-applied (replication-apply) transactions deliberately never drop on conflict — correct, there is no resume path for a dropped write. But the coordinated-retry branch recurses into `commit()` with **no `delay()` at all**, unlike the `ERR_BUSY` branch which backs off up to `MAX_RETRY_DELAY_MS` (1s). See [v5.2.1 `resources/DatabaseTransaction.ts#L658-L675`](https://github.com/HarperFast/harper/blob/v5.2.1/resources/DatabaseTransaction.ts#L658-L675) — identical on main.

Once `retries > 0`, every write in the transaction is fully re-saved per round (`#L496-L500`): entry reload + decode, full merge/index-diff/audit-construction/encode, fresh options object and promise chain per recursion. The long-transaction monitor explicitly exempts `sourceApply` (`#L994`), so nothing bounds or surfaces the spin.

## Impact

A sustained local writer on a hot replicated key can pin a worker thread in a zero-backoff allocation loop indefinitely (source-vs-source contention serializes via apply-loop backpressure; local-vs-source does not). While spun, the apply loop is wedged: all peer legs for that database pause holding their in-flight WS buffers (`replication_maxPayload`, default 100MB each), and the pause-stall watchdog eventually reconnects and redelivers the same conflict. Contributed to heap pressure in an internal 16-node cluster incident.

## Suggested fix

Add the same bounded backoff the `ERR_BUSY` branch already has to the coordinated-retry branch — one `delay(Math.min(this.retries * this.retries, MAX_RETRY_DELAY_MS))` before the recursion. Never-drop semantics are preserved; only the spin rate changes.

---
🤖 Investigated and filed by Claude (Fable) on Nathan's behalf

Contributor guide

Open the contributing guide

Research direction

Start in resources/DatabaseTransaction.ts at the coordinated-retry branch around lines 658-675, then compare it with the ERR_BUSY backoff and inspect the retry work around lines 496-500. Done means coordinated retries use the same bounded delay while preserving never-drop behavior; also review the sourceApply exemption around line 994.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, databases, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.