HarperFast / HarperFast/harper

Record locks Phase 1: rendezvous home ring, per-record delegations, and drain/recall

Open
#2,541 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of the Phase 1 record-lock redesign — the replacement for the Ricart–Agrawala arbitration rule
currently on #2498. Design note:
[`docs/record-lock-ownership.md`](https://github.com/HarperFast/harper/blob/feat/record-lock-phase1/docs/record-lock-ownership.md)
§§5, 6, 8. This is the arbitration rule itself: the piece that makes repeated locking by one node
cost zero cluster messages.

## What it is

Within a membership epoch (HarperFast/harper-pro#825), the arbiter for a key is a rendezvous hash
over `members[]` — its **home**. A node asks a key's home for a **delegation**: the exclusive right
to admit critical sections on that key for a bounded time. While a delegation is live,
`lock()`/`unlock()` are pure Phase 0 — the local rocksdb key lock, zero cluster messages. Releasing
the *application* lock does not release the delegation, so a node writing the same record repeatedly
pays one round and then nothing, and the delegate is in practice the last writer.

One arbiter per key is trivially exclusive, which is what deletes the entire grant state machine:
no deferral queues, no `(tsR, nodeName)` tiebreak, no synthesized grants, no split votes, no
contention-resolution protocol.

| | durable commits | frame deliveries | latency |
|---|---|---|---|
| Ricart–Agrawala (today, `P=12`) | 13 | 143 | slowest participant |
| First lock on a cold key | 0 | 2 unicast | 1 RTT to the home, 0 if local |
| Handoff to another node | 1 | `P−1` + 3 unicast | 2 RTT |
| Steady state | 0 | 0 | local key lock |

## Scope

- The home ring, the delegation table, and the wait queue (core).
- `transport.requestDelegation(...)` and a delegation server — unicast request/grant/recall over the
existing replication connections. `sendOperationToNode` is the existing precedent (harper-pro).
- **Ordered fencing tokens** (§5.1): `(epochNumber, homeIncarnation, delegationCounter)` with
`homeIncarnation` a durably persisted monotonic counter. A random incarnation makes a stale reply
identifiable but not *orderable*, and a home that restarts and re-issues generation 1 after having
issued generation 50 would let a delayed generation-50 write defeat its successor.
- **Lease transfer and local handle bounds** (§5.2): a handle may not outlive the delegation that
admitted it.
- **Drain must revoke capability, not just close the door** (§6). Recall has to invalidate live
handles, not merely stop issuing new ones; an in-flight native commit is the case the door-closing
version misses.
- **Aggregate caps** (§8): the branch's per-key and per-table caps exist to bound what a broadcast
round accumulates and are removed with it. Bound outstanding delegations per database and per
requester, bound expiration work per tick, cancel waiters. Eviction is asymmetric and the
asymmetry is the safety rule — a delegate may drop a delegation early, a **home may never forget
one before its expiry**.
- Failure containment as an executable contract: injected synchronous throws and rejections through
revoke, settlement callbacks, release persistence, shutdown and table removal. Callers settle,
admission stays closed, no fabricated handoff.
- Versioning the `recordLocks` capability so the two arbitration rules are mutually exclusive — a
cluster running both would have two independent arbiters for one key.

## Removed by this work

`LOCK_REQUEST`/`LOCK_GRANT` nibbles, per-peer round tracking, deferral queues, `(tsR, nodeName)`
ordering, synthesized grants, withdraw-on-timeout, and `agreedDown` DOWN-exclusion.

## Also fix here — three defects inherited with the substrate

A cross-model review of #2498 found these in code the redesign *keeps*, so they do not go away on
their own (§11 of the note):

- **Transport replacement does not fence live authority** (`resources/Table.ts`). Re-registering a
transport — a component reload is enough — closes the current coordinator and installs an empty
one without invalidating the handles the old one issued, so a successor can grant the same key
with no lease time elapsed.
- **The direct receive callback has no containment** (`resources/recordLockCoordinator.ts`). A
`LockUnavailableError` from coordinator construction escapes `deliverLockControlEntry()`; the
source-subscription sink already contains this and the direct callback must too.
- **The commit fence scans every write on every commit** (`resources/DatabaseTransaction.ts`). The
loop runs on ordinary transactions in core-only deployments that never register a transport, so a
bulk transaction with 100,000 plain writes pays 100,000 property checks. The transaction must
track whether it holds any lease-protected write and skip the pass when it does not — while still
fencing a released-but-staged locked write, which is the case the loop exists for.

## Blocked on

The measurement gate (HarperFast/harper-pro#824) and the epoch protocol (HarperFast/harper-pro#825).
The successor-freshness fence (#2542) is the sibling piece and lands with this one.

Refs #483

🤖 Filed by Claude Opus 5 on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start with docs/record-lock-ownership.md §§5, 6, and 8, then review resources/Table.ts, resources/recordLockCoordinator.ts, and resources/DatabaseTransaction.ts. Check the measurement gate #824, epoch protocol #825, and sibling freshness fence #2542 before implementation. Done means the delegation design, fencing, drain behavior, caps, failure containment, and capability versioning are implemented with the listed inherited defects fixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.