HarperFast / HarperFast/harper-pro
Record locks: a membership change is a ~6 minute cluster-lock outage; drain by recall instead of waiting out the delegation lease
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
Every membership change for a database's record-lock home map currently makes cluster locks **fully unavailable on every staged node for the whole drain interval — `DELEGATION_LEASE_MS + LOCK_LEASE_SKEW_MS`, 365 s** — and not only for the keys whose home is moving. Adding one node to a cluster is a ~6 minute cluster-lock outage for that database.
This is a documented, accepted tradeoff of the §4.3 transition (`replication/RECORD_LOCK_HOMES_DESIGN.md` → "The cost of this design, stated plainly", and that note's own reviewer section flags it as "worth a second look... more conservative than the core design doc's own prose implies is strictly necessary"). It is a hard blocker for enabling record locks by default (harper-pro#853) and close to disqualifying for a feature meant to be always-on.
## Why it happens
`record_lock_stage_generation` retracts `active` in the same durable write that records the staged generation (`planStage` writes `active: undefined`). That is deliberate and correct — it is what makes a successful stage *be* quiescence evidence, with no gap between "told about g+1" and "stopped granting under g". But `homeMap()` requires an `active` generation, so from that write until `record_lock_activate_generation`, every cluster-scoped `lock()` on that database answers 503 on that node.
The wait between the two exists so that authority already issued under `g` has expired before `g+1` can grant: a key whose home moves from A to a new node D would otherwise be grantable by D while a delegate still admits under A's grant. Rendezvous hashing moves roughly `1/(n+1)` of the keyspace on an addition, so additions are as exposed as removals.
## The fix: drain by recall, keep the timer as the fallback
The 365 s is a **timer**, chosen because you cannot recall what you cannot reach. For a *planned* transition every node is reachable — the operator is driving it — and core already has the exact mechanism:
> `recallDelegation(node, database, table, recall)` — "Home → delegate. Resolves once the delegate has drained and stopped admitting."
The home already drives this per grant in `#recall(grant)` (`resources/recordLockCoordinator.ts`), tracking `recalling` / `recallConfirmed` / `recallRetryAfterMono`. What is missing is a way to drive it for **every** outstanding grant at transition time and report completion.
With that, a staged node can prove quiescence instead of waiting for it: the window becomes the longest live critical section — typically milliseconds, bounded by the caller's own lease (`DEFAULT_LOCK_LEASE_MS` 30 s, `MAX_LOCK_LEASE_MS` 300 s) — rather than a flat 365 s. The timer stays as the fallback for a node that does not confirm, exactly as `record_lock_fence_external` is the fallback for a node that cannot be staged.
## Scope
**Core (harper), new entry point** — something of the shape:
```ts
/** Stop this node admitting under the current generation for `database`: recall every delegation it
* issued as a home, surrender every delegation it holds as a delegate, and resolve when all are
* confirmed — or report those still outstanding at the deadline. */
export async function quiesceDelegations(database: string, deadlineMs: number): Promise<{
recalled: number; surrendered: number; outstanding: { table: string; key: unknown; delegate?: string }[];
}>;
```
Both directions matter: a **delegate** is what admits, so this node must surrender what it holds; a **home** must recall what it issued so the delegates holding those stop. Reuse `#recall`'s existing state rather than adding a second recall path, and do not let a per-grant failure abort the sweep — collect it into `outstanding`.
**harper-pro:** call it from `stageGeneration` after the durable write (the write stops *new* grants; this drains the *existing* ones), bounded, and return the result in the stage response so an orchestrator can decide per node. A stage that reports `outstanding: []` on every node in `quiesce` means the operator may activate immediately with no wait.
## Acceptance
- A three-node cluster with live delegations stages a new generation and reports quiescence in well under a second, with no `DELEGATION_LEASE_MS` wait, and activates immediately afterward with no two-holder window — asserted by the existing exclusivity checks in `integrationTests/cluster/recordLockCluster.test.mjs`.
- A node that does not confirm its recalls reports them as `outstanding`, and the operator-facing guidance for that node falls back to the timer or `record_lock_fence_external`.
- The design note's "cost of this design" section is rewritten to state the new window and to keep the timer's role as the fallback.
## Not in scope
Narrowing the retraction to only the keys whose home moves (the "partial-quiesce protocol" the design note defers). That is the second, larger lever; with this one in place the outage is already bounded by a live critical section rather than a fixed interval.
Contributor guide
Research direction
Start with resources/recordLockCoordinator.ts and its existing #recall(grant) state, then trace stageGeneration and the durable stage write. Run integrationTests/cluster/recordLockCluster.test.mjs to understand the exclusivity checks. Done means harper-pro reports quiescence from both delegation directions, exposes outstanding recalls, avoids the fixed wait when all confirm, preserves the timer fallback, and updates replication/RECORD_LOCK_HOMES_DESIGN.md.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100