HarperFast / HarperFast/harper-pro
Record locks: one operator call to apply a home map across the cluster, given an explicit node list
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
Applying a record-lock home map is an N×2 loop today: `record_lock_stage_generation` on every node in `homes(g) ∪ homes(g+1)`, then `record_lock_activate_generation` on every node in `homes(g+1)`, with the operator carrying one identical `homes[]` list to each and deciding when the drain is done. `record_lock_propose_homes` (harper-pro#822) removed assembling the list by hand; it did not remove the loop.
This issue adds the single operator-facing call that drives the whole §4.3 transition, **given an explicitly supplied list of expected nodes**.
## Why the explicit list is what makes this safe
The rejected design in harper-pro#822 had a node *derive* the ring from `hdb_nodes`. That is unsafe for a specific, mechanical reason worth restating, because it is easy to re-propose: `homeMap()` iterates its **own** `active.homes`, so a node whose derived set is `[A]` checks no peers at all and serves its ring immediately. Two nodes with disjoint or merely incomplete views each get a usable map and both arbitrate the same key. **A digest cannot detect a participant omitted from the set being digested.**
When the operator states the set, there is nothing to omit. The list is complete by construction, so "ask every named node, fail if any does not answer" becomes a *complete* check rather than a best-effort one, and unreachable becomes a hard failure — which was not even expressible before. This operation may therefore orchestrate; it still may not derive.
## Scope
A new `requiresSuperUser` operation on `replication/recordLockHomes.ts`, e.g. `record_lock_apply_homes { database, homes[], generation? }`:
1. **Survey.** Ask every node in `homes` (and every node any of them reports in its current `active`/`staged` ring) for its current home-map state. Refuse unless **every** named node answers. Refuse if any node reports a ring containing a node the operator did not list — that is "you forgot a node that is still granting", and it is the residue of the second objection in #822 (no local state proves the absence of prior authority).
2. **Cross-check digests.** `digestOf(generation, canonicalHomes)` is deterministic over the canonicalized set; refuse unless what the nodes report is identical-or-absent.
3. **Stage everywhere**, in `homes(g) ∪ homes(g+1)`, with the one canonical list.
4. **Use the drain.** Each stage returns `quiesced` (harper-pro#856 / harper#2663). If `provesQuiescence()` holds for every node, activate immediately — no `DELEGATION_LEASE_MS + LOCK_LEASE_SKEW_MS` wait. Otherwise do **not** activate: return the per-node results and the earliest time activation would be safe, for a second call.
5. **Activate everywhere** in `homes(g+1)`. A departing node is staged and deliberately never activated.
6. **Report per node**, never a boolean.
## The two things that need designing, not just coding
**Auth for the hop.** `sendOperationToNode` connects with mTLS, so a peer authenticates it as a **node principal**, not `super_user` — and stage/activate are `requiresSuperUser`. Do **not** forward the operator's credentials per hop. Add a peer-callable operation authorized by the node principal, exactly as `record_lock_delegate` / `record_lock_recall` / `record_lock_barrier` already are (`recordLockRpc.ts`'s `principalNodeName`), whose handler **re-validates locally**: the receiving node checks the generation, the digest and its own row before writing. The node principal then relays a proposal each node independently verifies, rather than authorizing policy.
**Partial failure, which is why this is feasible at all.** `stage` retracts `active` in the same write, so a half-staged cluster has some nodes quiesced (their locks 503) and **none** double-granting — the failure direction is toward refusal. `planStage` and `planActivate` both have explicit idempotent noop branches for the same generation+digest, so the operator retries the same call. Specify the retry contract and make the response say exactly which nodes are in which state.
## Acceptance
- One call applies generation 1 to a fresh three-node cluster, and a cluster integration test asserts locks work afterwards with no manual per-node loop.
- One call performs a topology change, activating immediately when every node proves quiescence, and asserting the lock outage is bounded by the drain rather than by `DELEGATION_LEASE_MS`.
- A node that is unreachable, disagrees on the digest, or reports a ring member the operator did not list causes a refusal **before** anything is staged.
- A failure injected between stage and activate leaves the cluster in a safe, documented state, and re-running the same call completes the transition.
- The peer-callable hop rejects a caller that is not a node principal, and re-validates rather than trusting the payload.
## Not in scope
Deriving the list (`record_lock_propose_homes` remains the read-only helper). Narrowing the outage to only the keys whose home moves — the remaining lever on harper-pro#856.
## Dependencies
Branch from `feat/record-lock-cluster-transport` (harper-pro#822) — **none of this exists on `main`**. It also needs the drain from harper-pro#861, which is already merged into that branch.
Contributor guide
Assessment
This issue has not been assessed yet.