HarperFast / HarperFast/harper

Bound how long a sourceApply/isReplay transaction may hold write intents

Open
#2,495 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

#2471 asked for three things. #2473 ships the first two — the reporting surfaces — and deliberately defers the third, the reap policy. This issue carries that half so it is not lost when #2473 merges.

## What is still open

The open-transaction monitor exempts `sourceApply` / `isReplay` transactions from reaping. The exemption is deliberate: it exists so a write applied from a canonical source (a replication peer or an external caching source) is never dropped on a transient conflict. But it also means such a transaction can hold verification-table write intents indefinitely, parking other writers' commits, with no bound short of a process restart.

## Why the obvious fix is wrong

`abandonWrites()` is documented as barring any later commit, and a `sourceApply`/`isReplay` transaction has no resume path. Releasing its intents therefore drops the write while the cursor advances past it — permanent divergence, which is precisely why the exemption was created.

The only variant that preserves the write is abandon-then-replay onto a fresh transaction, and that cannot run while the transaction is parked in its pre-commit await: the writes it would replay are the ones whose blob file writes have not finished.

## What has to come first

#2471's production trigger is still unidentified. #2473 adds three log surfaces joined by the native transaction id, so the next occurrence should name the holder, its thread, its database and table, its staged-write count, and which state kept it from being reaped. That evidence should drive the policy rather than a mechanism chosen blind today.

## Acceptance

- A `sourceApply`/`isReplay` transaction cannot hold write intents unboundedly, **and** no canonical-source write is dropped or silently skipped as a result.
- Whatever mechanism is chosen states explicitly what happens to the replication cursor when it fires.

Refs #2471, #2450, #2473

Contributor guide

Open the contributing guide

Research direction

Start with the open-transaction monitor and the three reporting surfaces added by #2473, then inspect how sourceApply/isReplay transactions are exempted from reaping. Use the next occurrence to identify the holder and its state before choosing a policy. Done means write intents are bounded, canonical-source writes are preserved, and the replication-cursor behavior is explicit.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, databases, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.