HarperFast / HarperFast/harper-pro
Revoking the leader's admin mid-clone leaves the follower in terminal Unavailable — clone never retries (stale credential reused across leaderRequest calls)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## What happens
Revoking the leader's admin credential while a follower is mid-clone leaves the follower in a **terminal `Unavailable`** — not the crash loop hp#579 describes, and not hp#611's false-Available (fixed). The process stays alive, `get_status` stays reachable, and availability reports `Unavailable` unchanged from t=3.9s to t=103s with 0/40,000 keys copied. Nothing ever retries. Anyone triaging from hp#579's text would hunt the wrong signature (its documented failure is an uncaught 403 crash-looping the container).
## Mechanism (on `origin/main`)
Root cause: `cloneNode()` captures the leader credential **once** and reuses it unrefreshed across ~5 sequential `leaderRequest()` calls.
- `cloneSchemas()`'s `describe_all` 403s → the follower's `data` database is never pre-created (no subscription, no copy ever starts).
- `cloneJWTKeys()` then burns its fixed 3×250ms retry budget on the same dead token and throws.
- `cloneNode/cloneNode.ts` — `finishCloneSetup()`'s catch around `cloneJWTKeys()` (~lines 508–530) explicitly sets `{id:'availability', status:'Unavailable'}`, clears `cloned`, and returns `false`; that trips `if (!(await finishCloneSetup())) return;` in `cloneNode()`'s first try block, returning from the whole function **before** `monitorSync()` (line 346) is ever reached. The one-shot boot script has no other retry path; the server (already started by the earlier `main()`) just sits at the last published availability.
## Repro
Two-node clone; revoke the leader-side admin user after the follower's initial connection but before schema clone completes. Follower wedges as described; only operator intervention (restart with valid credentials) recovers.
## Existing work checked
hp#579 (OPEN — crash-loop signature, different outcome of the same scenario family; this issue narrows/corrects its framing), hp#611 (CLOSED 2026-08-06 via #650/#657/#661 — false-Available, unrelated mechanism). No open/closed harper-pro issue matches "terminal Unavailable", `finishCloneSetup`, or `cloneJWTKeys`.
## What this does not prove
Whether a *refreshed* credential mid-clone recovers cleanly (the fix decides retry-with-refresh vs fail-fast-and-exit so orchestration restarts); behavior when revocation lands after `monitorSync` starts.
---
*From dispatch QA finding F-207, verified against harper-pro origin/main, 2026-08-29.*
Contributor guide
Research direction
Start in cloneNode/cloneNode.ts, especially cloneNode(), finishCloneSetup(), cloneSchemas(), cloneJWTKeys(), and monitorSync(). Reproduce the two-node clone with the leader credential revoked during schema cloning, then trace how the captured credential flows through leaderRequest() calls. Done should include a regression scenario and a defined recovery outcome instead of leaving the follower terminally Unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, typescript
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100