HarperFast / HarperFast/harper

Record locks: stronger `lock()` guarantees than exclusion-only (fenced ordering, quorum-confirmed writes)

Open
#2,540 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

`lock()`'s cluster guarantee was decided on 2026-09-09 as **exclusion-only** (option (b) of
[`docs/record-lock-ownership.md`](https://github.com/HarperFast/harper/blob/feat/record-lock-phase1/docs/record-lock-ownership.md)
§7.3). This issue records the two arms that were *not* taken, so the limitations they leave are
tracked rather than only documented.

## What exclusion-only promises

At most one node admits a critical section for `(database, table, key)` at a time, and a node
admitted after a clean handoff has applied its predecessor's committed writes to that key before it
admits. Conflict resolution stays last-write-wins, unchanged for every write in the database.

## The two limitations it leaves

**Neither is crash-only, and both are reachable on a completely clean handoff** — (1b) below by
construction, and (2) whenever the key's home has lost the handoff's dependency set to a restart, an
epoch change, or cap eviction.

1. **A predecessor's write can outrank its successor's under last-write-wins.** LWW compares the
transaction timestamp assigned when the write was *staged* (`resources/Table.ts:3113`), and lease
expiry orders admissions, not timestamps. **`lock()` changes nothing about conflict resolution** —
a locked write and an unlocked write resolve identically, by timestamp, per field, with CRDT ops
folded and the surviving shape depending on both writes' shapes — so whatever the pair would have
done to the record without a lock is what they do with one, silently and with no error raised.
Two routes in, and they fail differently:
- **(1a) clock skew.** The predecessor's clock ran ahead of its successor's and its write is still
in flight when the successor writes.
- **(1b) a timestamp pushed ahead on purpose, on a clean handoff.** A caller-supplied future
`context.timestamp`, or a mixed explicit transaction whose later timestamp becomes the handle's
floor — both deliberate Phase 0 behavior (`DESIGN.md`, "Dual-clock"). The predecessor commits,
replicates, drains and releases cleanly; the successor is admitted with correct freshness, reads
the current value, writes at real wall-clock time, and **loses silently** because the stored
version is stamped in the future. No crash, no skew, nothing in flight. The only general defense
today is not to stamp future timestamps under a lock.

2. **Successor freshness is not promised on the recovery path.** The recovery barrier drains inbound
streams from every *reachable* member, so a write the predecessor committed and did not replicate
is invisible to it: the barrier succeeds and the successor reads a stale value. Nothing failed, so
failing closed does not detect it. Three routes, only the first a crash — the predecessor crashed;
it is unreachable; or its native commit passed the pre-commit expiry fence and settles *after* the
barrier was measured. The third needs no failure beyond a slow commit. **Two damaged effects on
the one record**: the predecessor's transaction is dropped by LWW, and the successor's own write is
computed from the stale value it read — stored balance 150, successor reads 100 and writes 80, so
the surviving value is wrong on its own terms and not merely stale. That is exactly the
read-then-conditionally-write `lock()` exists for.

Also worth stating because the API docs will be written from it: §2's exclusion invariant covers a
predecessor that can still **admit or commit**. What ships excludes admission; the commit half holds
only up to the pre-submission expiry fence (`resources/DatabaseTransaction.ts:1213`), and a native
commit that clears that fence and settles afterwards is limitation 2's third route.

## The two arms that close them

They are independent axes and can be taken separately.

### (a) Fenced conflict ordering — closes limitation 1

Every write carries a fencing generation (0 when unlocked) and conflict resolution becomes
`(generation, timestamp, origin)` lexicographic. Total, no cycle. The reason it was not taken is
that its cost falls on deployments that never call `lock()`:

- It changes `_writeUpdate`'s resolution rule for **all** writes and adds a field to the record
contract — a stored-format and older-binary compatibility decision, including what happens if the
feature is disabled after fenced records exist.
- It **reverses the documented Phase 0 relationship with ordinary writes** (`DESIGN.md`, "Phase 0
contract"): an ordinary update landing after a fenced value would permanently lose, regardless of
age.
- Older writes enter resequencing and merge logic and deletes take a separate path, so a generation
must be honored by patches and deletes too.
- A generation alone has no **rejection floor that survives tombstone reclamation**: generation 1
puts, generation 2 deletes, the tombstone is reclaimed, a delayed generation-1 put resurrects the
record. Fenced mode owes a retained floor across tombstone cleanup, snapshot copy, restore and
replay.

Taking this arm also adds a test obligation: stale full writes, patches and deletes against
successor records and reclaimed tombstones, asserting values *and* indices.

### (c) Quorum-confirmed locked writes — closes limitation 2

A locked write is not considered settled until it has replicated to a majority, so a successor's
barrier over reachable members necessarily sees it. This is the only arm that makes the freshness
invariant hold in recovery mode as well as on clean handoff. `X-Replicate-To: N;confirm=M`
(`server/REST.ts:265`) is the nearest existing mechanism, but this arm cannot simply expose it.

- **It is not reachable by an application caller.** `X-Replicate-To` / `confirm=` is super-user-gated
— `checkContextPermissions` (`resources/Table.ts:7258`) throws 403 otherwise — so there is no
caller-side mitigation for limitation 2 today at all.
- **It is entangled with residency.** A numeric `X-Replicate-To` sets residency to `[self, ...N
nodes]` and truncates existing residency on update (`getResidency`, `resources/Table.ts:1574`);
`*` leaves `replicateTo` undefined and falls back to the database's configured
`replication.replicateTo` count (`:1578`, `:642`) or to `setResidencyById`. In a twelve-node cluster
configured `replicateTo: 3`, either form leaves the record on four nodes, and a successor's barrier
over the other eight reachable members satisfies without any of them ever having held the locked
write — the opposite of what the barrier needs.
- **The read side is missing.** Intersection needs `M + reachable > |cluster|`, and the recovery
barrier has no read-quorum floor: it drains every *reachable* member and deliberately tolerates
unreachable ones. So the barrier must also require a majority and fail closed below it. Forcing that
intersection costs any single down node blocking every locked write — the availability property this
whole redesign exists to avoid — and `M` is a literal that silently under-confirms after a
scale-out.
- **Write confirmation cannot reach the late-settling commit at all.** That predecessor's commit had
not settled when the barrier was measured, so no confirmation level makes a barrier taken at `T`
observe a commit that replicates at `T+δ`. That route needs the commit *fenced*, not confirmed — so
§2's invariant as written needs **both** arms, (a) and (c), not (c) alone.

Arm (c) does not touch limitation 1 on its own: a quorum-confirmed write is more *visible*, and
visibility is not ordering.

## Why this is deferred rather than rejected

`X-Replicate-To: N;confirm=M` already exists as a caller-side lever that narrows (2), route (1b) has
a discipline that avoids it (do not stamp future timestamps under a lock), and the base feature has
no measurements yet
— option (a) in particular puts a stored-format decision on the critical path of a feature whose
cost nobody has measured. Revisit once the Phase 1 protocol is running and there is a workload that
needs the stronger promise.

Design note: [`docs/record-lock-ownership.md`](https://github.com/HarperFast/harper/blob/feat/record-lock-phase1/docs/record-lock-ownership.md) §7.3, §10.

Refs #483

🤖 Filed by Claude Opus 5 on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start with docs/record-lock-ownership.md §7.3 and §10, then read the Phase 0 “Dual-clock” section in DESIGN.md. Inspect resources/Table.ts, resources/DatabaseTransaction.ts, and server/REST.ts at the referenced locations to understand the existing ordering, expiry, replication, and permission paths. No implementation is requested now; done is a measured workload and a design decision about whether to pursue fenced ordering, quorum confirmation, or both.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.