HarperFast / HarperFast/harper
Bound how long a sourceApply/isReplay transaction may hold write intents
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
#2471 asked for three things. #2473 ships the first two — the reporting surfaces — and deliberately defers the third, the reap policy. This issue carries that half so it is not lost when #2473 merges.
## What is still open
The open-transaction monitor exempts `sourceApply` / `isReplay` transactions from reaping. The exemption is deliberate: it exists so a write applied from a canonical source (a replication peer or an external caching source) is never dropped on a transient conflict. But it also means such a transaction can hold verification-table write intents indefinitely, parking other writers' commits, with no bound short of a process restart.
## Why the obvious fix is wrong
`abandonWrites()` is documented as barring any later commit, and a `sourceApply`/`isReplay` transaction has no resume path. Releasing its intents therefore drops the write while the cursor advances past it — permanent divergence, which is precisely why the exemption was created.
The only variant that preserves the write is abandon-then-replay onto a fresh transaction, and that cannot run while the transaction is parked in its pre-commit await: the writes it would replay are the ones whose blob file writes have not finished.
## What has to come first
#2471's production trigger is still unidentified. #2473 adds three log surfaces joined by the native transaction id, so the next occurrence should name the holder, its thread, its database and table, its staged-write count, and which state kept it from being reaped. That evidence should drive the policy rather than a mechanism chosen blind today.
## Acceptance
- A `sourceApply`/`isReplay` transaction cannot hold write intents unboundedly, **and** no canonical-source write is dropped or silently skipped as a result.
- Whatever mechanism is chosen states explicitly what happens to the replication cursor when it fires.
Refs #2471, #2450, #2473
Contributor guide
Research direction
Start with the open-transaction monitor and the three reporting surfaces added by #2473, then inspect how sourceApply/isReplay transactions are exempted from reaping. Use the next occurrence to identify the holder and its state before choosing a policy. Done means write intents are bounded, canonical-source writes are preserved, and the replication-cursor behavior is explicit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100