HarperFast / HarperFast/harper-pro

Revoking the leader's admin mid-clone leaves the follower in terminal Unavailable — clone never retries (stale credential reused across leaderRequest calls)

Open
#785 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## What happens

Revoking the leader's admin credential while a follower is mid-clone leaves the follower in a **terminal `Unavailable`** — not the crash loop hp#579 describes, and not hp#611's false-Available (fixed). The process stays alive, `get_status` stays reachable, and availability reports `Unavailable` unchanged from t=3.9s to t=103s with 0/40,000 keys copied. Nothing ever retries. Anyone triaging from hp#579's text would hunt the wrong signature (its documented failure is an uncaught 403 crash-looping the container).

## Mechanism (on `origin/main`)

Root cause: `cloneNode()` captures the leader credential **once** and reuses it unrefreshed across ~5 sequential `leaderRequest()` calls.

- `cloneSchemas()`'s `describe_all` 403s → the follower's `data` database is never pre-created (no subscription, no copy ever starts).
- `cloneJWTKeys()` then burns its fixed 3×250ms retry budget on the same dead token and throws.
- `cloneNode/cloneNode.ts` — `finishCloneSetup()`'s catch around `cloneJWTKeys()` (~lines 508–530) explicitly sets `{id:'availability', status:'Unavailable'}`, clears `cloned`, and returns `false`; that trips `if (!(await finishCloneSetup())) return;` in `cloneNode()`'s first try block, returning from the whole function **before** `monitorSync()` (line 346) is ever reached. The one-shot boot script has no other retry path; the server (already started by the earlier `main()`) just sits at the last published availability.

## Repro

Two-node clone; revoke the leader-side admin user after the follower's initial connection but before schema clone completes. Follower wedges as described; only operator intervention (restart with valid credentials) recovers.

## Existing work checked

hp#579 (OPEN — crash-loop signature, different outcome of the same scenario family; this issue narrows/corrects its framing), hp#611 (CLOSED 2026-08-06 via #650/#657/#661 — false-Available, unrelated mechanism). No open/closed harper-pro issue matches "terminal Unavailable", `finishCloneSetup`, or `cloneJWTKeys`.

## What this does not prove

Whether a *refreshed* credential mid-clone recovers cleanly (the fix decides retry-with-refresh vs fail-fast-and-exit so orchestration restarts); behavior when revocation lands after `monitorSync` starts.

---
*From dispatch QA finding F-207, verified against harper-pro origin/main, 2026-08-29.*

Contributor guide

Open the contributing guide

Research direction

Start in cloneNode/cloneNode.ts, especially cloneNode(), finishCloneSetup(), cloneSchemas(), cloneJWTKeys(), and monitorSync(). Reproduce the two-node clone with the leader credential revoked during schema cloning, then trace how the captured credential flows through leaderRequest() calls. Done should include a regression scenario and a defined recovery outcome instead of leaving the follower terminally Unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.