HarperFast / HarperFast/harper

A write from an aborted replicated transaction can commit on its own when another key's load fails

Open
#2,237 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

When a replicated transaction is applied and one record's resource load rejects, the transaction aborts — but a *different* key's write, still suspended on its own slow load, can resume afterwards and reach `addWrite()` on the now-closed transaction, where `save()` commits it **alone, with its own audit entry**. The follower durably applies part of a leader transaction that never landed as a whole.

Pre-existing. harper#2235 narrows it (same-key successors now short-circuit when their predecessor rejects) but cannot close it, because the surviving write is under a different key.

## Mechanism (confirmed in source)

The apply loop dispatches each record's `writeUpdate()` without awaiting it, and `writeUpdate()` suspends on `await Table.getResource(...)`. For a leader transaction `{put A, delete B}`:

1. A's `getResource` rejects. The rejection propagates through `Promise.all(writePromises)` at `end_txn` and the transaction aborts.
2. B's load was slow. Its continuation resumes after the abort and calls `_writeDelete` → `txnForContext(context)` → the context's transaction is closed.
3. In `DatabaseTransaction.save()`, a closed transaction takes the `immediateCommit` branch: a fresh `RocksTransaction` is created and committed on the spot. B lands, with its own audit entry.

## Two different outcomes depending on the write type

This is worth stating because the two are opposite failures and only one is the "escape":

- **Eagerly-saved write types** (`delete`, `invalidate`, `relocate`, `publish`) — `addWrite` calls `save()`, which commits them alone. **A write from an aborted transaction becomes durable.**
- **`put`/`patch`** — `deferSave: true`, so `addWrite` does not call `save()` and nothing else does either. The write is staged into a transaction nobody ever commits and is **silently discarded**. Observed while testing harper#2235: in that configuration neither write commits and no error surfaces.

So the same race either over-applies or under-applies, depending on which write survives.

## Impact

Follower diverges from leader: it holds a delete the leader's transaction rolled back, or drops a put that the leader committed. Nothing alarms — the apply loop logs the original failure and carries on, and the divergence persists until a full copy or a later write to that key rewrites it.

## Suggested fix

An `open !== TRANSACTION_STATE.OPEN` check in `writeUpdate`'s continuation, so a write whose transaction is already gone is dropped rather than committed alone. That is a small change, but it decides drop-vs-commit semantics for **every** source-applied write, so it wants its own tests — including the `put` case above, where the current behavior is already a silent drop.

Documented in `DESIGN.md` under the repeat-writes section as part of harper#2235.

Contributor guide

Open the contributing guide

Research direction

Start at the replicated apply loop and follow writeUpdate through Table.getResource and DatabaseTransaction.save, using the cross-key failure sequence in the issue. Review DESIGN.md's repeat-writes section, then add regression coverage for surviving delete and put writes after another key's load rejects. Done means neither write commits outside the aborted transaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.