HarperFast / HarperFast/harper-pro

Record locks: one operator call to apply a home map across the cluster, given an explicit node list

Open
#862 0 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

Applying a record-lock home map is an N×2 loop today: `record_lock_stage_generation` on every node in `homes(g) ∪ homes(g+1)`, then `record_lock_activate_generation` on every node in `homes(g+1)`, with the operator carrying one identical `homes[]` list to each and deciding when the drain is done. `record_lock_propose_homes` (harper-pro#822) removed assembling the list by hand; it did not remove the loop.

This issue adds the single operator-facing call that drives the whole §4.3 transition, **given an explicitly supplied list of expected nodes**.

## Why the explicit list is what makes this safe

The rejected design in harper-pro#822 had a node *derive* the ring from `hdb_nodes`. That is unsafe for a specific, mechanical reason worth restating, because it is easy to re-propose: `homeMap()` iterates its **own** `active.homes`, so a node whose derived set is `[A]` checks no peers at all and serves its ring immediately. Two nodes with disjoint or merely incomplete views each get a usable map and both arbitrate the same key. **A digest cannot detect a participant omitted from the set being digested.**

When the operator states the set, there is nothing to omit. The list is complete by construction, so "ask every named node, fail if any does not answer" becomes a *complete* check rather than a best-effort one, and unreachable becomes a hard failure — which was not even expressible before. This operation may therefore orchestrate; it still may not derive.

## Scope

A new `requiresSuperUser` operation on `replication/recordLockHomes.ts`, e.g. `record_lock_apply_homes { database, homes[], generation? }`:

1. **Survey.** Ask every node in `homes` (and every node any of them reports in its current `active`/`staged` ring) for its current home-map state. Refuse unless **every** named node answers. Refuse if any node reports a ring containing a node the operator did not list — that is "you forgot a node that is still granting", and it is the residue of the second objection in #822 (no local state proves the absence of prior authority).
2. **Cross-check digests.** `digestOf(generation, canonicalHomes)` is deterministic over the canonicalized set; refuse unless what the nodes report is identical-or-absent.
3. **Stage everywhere**, in `homes(g) ∪ homes(g+1)`, with the one canonical list.
4. **Use the drain.** Each stage returns `quiesced` (harper-pro#856 / harper#2663). If `provesQuiescence()` holds for every node, activate immediately — no `DELEGATION_LEASE_MS + LOCK_LEASE_SKEW_MS` wait. Otherwise do **not** activate: return the per-node results and the earliest time activation would be safe, for a second call.
5. **Activate everywhere** in `homes(g+1)`. A departing node is staged and deliberately never activated.
6. **Report per node**, never a boolean.

## The two things that need designing, not just coding

**Auth for the hop.** `sendOperationToNode` connects with mTLS, so a peer authenticates it as a **node principal**, not `super_user` — and stage/activate are `requiresSuperUser`. Do **not** forward the operator's credentials per hop. Add a peer-callable operation authorized by the node principal, exactly as `record_lock_delegate` / `record_lock_recall` / `record_lock_barrier` already are (`recordLockRpc.ts`'s `principalNodeName`), whose handler **re-validates locally**: the receiving node checks the generation, the digest and its own row before writing. The node principal then relays a proposal each node independently verifies, rather than authorizing policy.

**Partial failure, which is why this is feasible at all.** `stage` retracts `active` in the same write, so a half-staged cluster has some nodes quiesced (their locks 503) and **none** double-granting — the failure direction is toward refusal. `planStage` and `planActivate` both have explicit idempotent noop branches for the same generation+digest, so the operator retries the same call. Specify the retry contract and make the response say exactly which nodes are in which state.

## Acceptance

- One call applies generation 1 to a fresh three-node cluster, and a cluster integration test asserts locks work afterwards with no manual per-node loop.
- One call performs a topology change, activating immediately when every node proves quiescence, and asserting the lock outage is bounded by the drain rather than by `DELEGATION_LEASE_MS`.
- A node that is unreachable, disagrees on the digest, or reports a ring member the operator did not list causes a refusal **before** anything is staged.
- A failure injected between stage and activate leaves the cluster in a safe, documented state, and re-running the same call completes the transition.
- The peer-callable hop rejects a caller that is not a node principal, and re-validates rather than trusting the payload.

## Not in scope

Deriving the list (`record_lock_propose_homes` remains the read-only helper). Narrowing the outage to only the keys whose home moves — the remaining lever on harper-pro#856.

## Dependencies

Branch from `feat/record-lock-cluster-transport` (harper-pro#822) — **none of this exists on `main`**. It also needs the drain from harper-pro#861, which is already merged into that branch.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.