HarperFast / HarperFast/harper-pro
Clone bootstrap races cluster onboarding: follower auths as HDB_ADMIN after leader deletes it, permanent 403 crash loop
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
Multi-instance Fabric-hosted cluster creation races against the cluster's own onboarding hardening step. The **follower node's clone bootstrap authenticates to the leader as `HDB_ADMIN`**, but Fabric onboarding **deletes the leader's `HDB_ADMIN` user ~90 seconds after creation** (immediately after provisioning the real org user), independent of whether any follower has cloned yet. If the follower's first clone request arrives after that deletion — which is close to guaranteed given normal follower-side setup time (DNS, image pull, cert setup, storage, symphony routing) — the clone fails with a permanent, unrecoverable 403 and the follower container crash-loops forever.
## Evidence (cluster `alpha5`, dev, 2026-07-14)
Audit log on the leader instance (`system.hdb_user`), in order:
```
12:07:05.7 insert HDB_ADMIN created (bootstrap super_user)
12:07:19.4 update HDB_ADMIN gets a refresh_token ← CM mints the clone-auth JWT for the follower here
12:08:56.3 insert kris@harperdb.io created (real org user, Fabric onboarding)
12:08:56.5 delete HDB_ADMIN deleted ← ~90s after creation, ~0.2s after the real user lands
```
The follower's compose file embeds an `HDB_LEADER_TOKEN` (JWT, `username: HDB_ADMIN`, `super_user: true`, minted at 12:07:19, valid until 12:22:19 — 15 min TTL). The follower's container log shows a permanent crash loop:
```
Starting clone node from leader: https://:9925 using credential auth
Error: Leader request failed: 403 Forbidden
at leaderRequest (dist/cloneNode/cloneNode.js:984:15)
at async cloneNode (dist/cloneNode/cloneNode.js:232:9)
```
Leader-side log for the same requests:
```
[main/0] [warn] [operation]: User 'HDB_ADMIN' is not permitted to get_status
[main/0] [warn] [operation]: User 'HDB_ADMIN' is not permitted to add_node_back
```
The JWT itself verifies fine (correctly signed, not expired — confirmed by comparing container clock to token `iat`/`exp`). The rejection is a live permission/role-lookup failure: once `HDB_ADMIN` is deleted, any operation requiring a live role resolution (`get_status`, `add_node_back` — both registered via the dynamic `server.registerOperation` path in `replication/setNode.ts`) permanently denies, even though the raw token still parses as valid. `cloneNode.ts`'s `leaderRequest()` (harper-pro `cloneNode/cloneNode.ts:1061`) never re-mints or refreshes credentials on failure — it retries the identical stale token forever, so this is not a transient race that self-heals; it is a permanent dead end once hit.
Confirmed directly on the leader via `list_users`: only `kris@harperdb.io` exists; no `HDB_ADMIN` row (matching the audit log's `delete` entry).
## Impact
Any Fabric-hosted cluster with 2+ instances is at risk of this: the follower needs to complete DNS + image pull + cert/storage/symphony setup and reach its *first* leader clone request within the leader's onboarding window (~90s in this sample) or the cluster is permanently stuck with the follower crash-looping. Slower image pulls, colder hosts, or heavier instance setup make this more likely to hit, not less.
## Where to look
- `harper-pro` `cloneNode/cloneNode.ts` — `leaderRequest()` (~line 1061) authenticates with a single credential/token captured at container start and never refreshes it; `cloneNode()` (~line 232) doesn't distinguish "leader not ready yet" from "credential permanently invalid" and retries identically either way.
- `harper-pro` `replication/setNode.ts` — `get_status`/`add_node_back` registered via `server.registerOperation`; worth confirming whether the dynamic-registration authorization path (`utility/operation_authorization.ts`, see the `registerGrantableOperation`/"gate-2" comments in `components/secretOperations.ts`) is correctly bypassing for a token's `super_user` claim vs. requiring a live DB role lookup that fails once the user is deleted.
- Whichever component performs Fabric's post-create onboarding (creates the real org user + deletes `HDB_ADMIN` on the leader instance) — not found in `central-manager`'s own repo; likely in `studio`/`fabric_ui` calling the instance's operations API directly, or elsewhere in `central-manager` under a name not grepped here. Needs to be located.
## Possible directions (not evaluated/decided — needs owner judgment)
1. Delay deleting the leader's `HDB_ADMIN` until all instances in the cluster have finished cloning (onboarding-side fix).
2. Have follower-side clone bootstrap use a credential that isn't scheduled for deletion (e.g. a dedicated clone-bootstrap service account, or cert-based auth instead of `HDB_ADMIN` credential/token auth).
3. Have `cloneNode.ts`'s `leaderRequest()` distinguish a genuine permission/auth failure from "not ready yet" and fetch a fresh credential (e.g. re-request from CM) instead of retrying a token that can never succeed.
## Repro
Create any 2+ instance Fabric-hosted cluster on dev (e.g. via Studio) and watch the follower instance's container logs; check the leader's `read_audit_log` for `system.hdb_user` to see the `HDB_ADMIN` create/delete timing relative to the follower's first clone attempt.
Contributor guide
Research direction
Reproduce with a 2+ instance Fabric-hosted cluster and compare follower logs with the leader's read_audit_log output. Read cloneNode/cloneNode.ts around cloneNode() and leaderRequest(), then inspect replication/setNode.ts and the referenced authorization paths; locate the Fabric onboarding code that deletes HDB_ADMIN. Done means follower cloning completes after onboarding without a permanent 403 crash loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100