HarperFast / HarperFast/harper
Multi-key transaction reads are not atomic to concurrent readers (cross-key visibility)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A concurrent reader can observe a **partially-applied committed multi-key transaction** — e.g. in a transfer that debits A and credits B in one request `transaction()`, a reader sometimes sees the credit before the matching debit. Cross-key **commit visibility is not atomic** to concurrent readers. Eventual conservation always holds (no data loss); it's the visibility window during commit.
## Severity
Medium — no data loss/corruption (the invariant always reconciles), but an app relying on multi-key atomic reads (transfers, cross-row invariants) can read transient inconsistent states. This is really an **isolation-contract question**: does Harper promise cross-key read-atomic visibility for a single request transaction?
## Repro (self-contained)
`Account` table seeded so the total across accounts is a fixed invariant (e.g. 8 accounts × 1000 = 8000). A custom resource `Transfer` that, in one request `transaction()`, moves 1 unit A→B. Then:
- fire many concurrent overlapping transfers (`Promise.all`, multiple rounds), and
- run a concurrent reader that reads the pair (A,B) repeatedly — via two sequential `get()`s **and** via a single `search()` snapshot — checking each read-pair sums to the invariant.
Observed: the reader over-counts (pair sum transiently 2001/2002/2003 = credit visible before debit) in **every** cell — both engines (RocksDB + LMDB), single- and multi-worker, and **the single-`search()` snapshot reader tears too** (so it's not a per-`get()` snapshot artifact). Final/total conservation is always correct; no deadlock (opposite-order key locking resolves cleanly, 0 timeouts).
Confirmed general across BOTH write paths:
- atomic deltas (`addTo`/`subtractFrom`) — plausibly by-design (independent per-key CRDT merges), AND
- **plain `set()` writes inside one `transaction()`** — the decisive case: the torn read persists, so this is not a CRDT-delta quirk.
## Mechanism
The keys of one transaction become visible to concurrent readers at slightly different points rather than as a single atomic commit-visibility step. (Contrast: cross-table commit/rollback *durability*-atomicity is correct — a mid-handler throw rolls back all tables. This is the distinct, weaker *visibility*-atomicity property.)
## Recommendation
Settle the isolation contract:
- If Harper intends multi-key read-atomic visibility → make commit visibility atomic across a transaction's keys (read-visibility fix).
- If not → document the isolation level loudly (single-key atomic; multi-key not read-atomic) so apps model invariants accordingly.
Ties the broader concurrency-primitive gap (atomic single-key delta is the only safe concurrent primitive; CAS-shaped idioms are not).
---
*Surfaced by the QA-explorer campaign (QA-139 delta path + QA-142 plain-write tie-breaker) against Harper `7aaa5a152`. The transaction/visibility code paths are byte-identical to `main` @`6797f091d` (the only feature-branch delta is an unrelated replication-revalidation line). Ready-to-promote regression test exists. Filed by Claude (Opus 4.8) for @kris.*
Contributor guide
Research direction
Start by reproducing the concurrent transfer scenario using transaction(), get(), and search(), then compare the addTo()/subtractFrom() and set() write paths across the RocksDB and LMDB engines. The issue does not name files or a test path; done requires settling the read-visibility contract and adding the referenced regression coverage or documentation accordingly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100