Staged component deploys: land build-aside-then-swap as a sequence of small PRs

Coordination issue for landing staged (build-aside-then-swap) component deploys as a sequence of small, independently mergeable PRs. This replaces the single large effort in #1849 / #2301, both of which become reference-only.

## The destination

Where this sequence is carrying us, as one picture. Colour is **status, not importance** —
what has already landed, what is next, and what may never be built at all:

```mermaid
flowchart TD
L1["MERGED
on main today"]:::shipped
L2["IN PROGRESS
being written now"]:::progress
L3["PLANNED
next up, not written yet"]:::planned
L4["SHELVED
maybe one day, maybe never"]:::contested
L5["NOT A STEP
what the operator asks for"]:::actor

L1 ~~~ L4
L2 ~~~ L5

classDef shipped fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
classDef progress fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e3a8a
classDef planned fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#78350f
classDef contested fill:#fee2e2,stroke:#dc2626,stroke-width:2px,stroke-dasharray:5 4,color:#7f1d1d
classDef actor fill:#f1f5f9,stroke:#64748b,stroke-width:1px,color:#0f172a
```

```mermaid
flowchart TD
OP["Operator issues deploy_component
package: / ref: / payload"]:::actor

subgraph ORIGIN["One node resolves — step 7"]
R1["Resolve and pack the artifact once,
hashed and sized on the way through"]:::planned
R2["hdb_deployment row carries payload_blob
— the bytes every node will install.
Shipped today, for payload deploys"]:::shipped
end

subgraph NODE["Every node installs those same bytes"]
N1["Install from the replicated blob,
no independent resolution per node"]:::planned
N2["Build aside under .deploy-staging/id —
the live tree is never a construction site"]:::shipped
N3["Certify the candidate by loading it
for real, off the serving path
step 2 — premise contested"]:::contested
N4["Activate as one transaction: live moves aside,
candidate takes its place, and the root-config
entry is published as an effect of that swap
step 1 landed the swap, step 3 adds the config"]:::planned
N5["Discard the candidate — live tree and
root config untouched, and the operation
reports the failure it actually hit"]:::shipped
N6["Dormant builds kept under a bound
step 4"]:::shipped
N7["Previous live tree kept as the revert target
step 5"]:::planned
end

RE["deploy_component by deployment id —
activate a build that is already staged
step 6"]:::progress
RV["revert_component — addressed,
idempotent, resumable
step 5"]:::planned

OP --> R1 --> R2
R2 -- "whole-operation replication" --> N1
N1 --> N2 --> N3
N3 -- "pass" --> N4
N3 -- "fail, or no verdict" --> N5
N4 --> N6
N4 --> N7
N6 --> RE
N7 --> RV
RE -.-> N4
RV -.-> N4

classDef shipped fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
classDef progress fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e3a8a
classDef planned fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#78350f
classDef contested fill:#fee2e2,stroke:#dc2626,stroke-width:2px,stroke-dasharray:5 4,color:#7f1d1d
classDef actor fill:#f1f5f9,stroke:#64748b,stroke-width:1px,color:#0f172a
```

Read it as a single claim: **one node resolves, every node installs those same bytes, and no node's live tree changes until a candidate exists, has been checked, and can be swapped in as one transaction that includes the root-config entry.** Everything hanging off the bottom — retention, revert, deploy-from-an-aside — exists to make that transaction reversible after the fact.

The same destination, and the same colours, from the point of view of one component's directory on disk:

```mermaid
stateDiagram-v2
direction TB
[*] --> Staged: deploy_component builds aside
Staged --> Certified: load verdict passes
Staged --> Discarded: build fails, or no verdict
Certified --> Live: activate — two renames plus root config, journalled
Certified --> Dormant: staged only, activation deferred
Dormant --> Live: deploy_component by deployment id
Dormant --> [*]: evicted by stagingRetention.maxCount
Live --> Retired: superseded by a later activation
Retired --> Live: revert_component
Retired --> [*]: aged out
Discarded --> [*]

Staged: STAGED — extracted and installed under .deploy-staging/id
Certified: CERTIFIED — .complete minted only from a real load verdict
Live: LIVE — serving, and named by root config
Dormant: DORMANT — built, never activated, bounded
Retired: RETIRED — previous tree, kept so a revert has somewhere to go
Discarded: DISCARDED — nothing outside .deploy-staging was touched

note right of Certified
A crash is not a state of its own.
Boot recovery reads the markers and the journal
and finishes the transition, forward or back.
end note

classDef shipped fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
classDef planned fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#78350f
classDef contested fill:#fee2e2,stroke:#dc2626,stroke-width:2px,stroke-dasharray:5 4,color:#7f1d1d
classDef actor fill:#f1f5f9,stroke:#64748b,stroke-width:1px,color:#0f172a

class Staged,Dormant,Discarded shipped
class Certified contested
class Retired planned
class Live actor
```

**Deliberately outside the picture:**

- **Cross-node activation ordering (#2294).** Build-once makes every node install identical bytes; it does not *order* two concurrently-originated deploys, so nodes can still settle on different versions with both operations reporting success. It is also the only part that needs multi-node test infrastructure we do not have.
- **The pathname gap.** Activation is still two renames, so the live pathname is briefly absent and a component that opens its own files during a request can observe it. Closing it needs immutable version directories plus an atomically-swapped locator — the symlink and in-memory-cutover successors below, neither scheduled.
- **Whether certification belongs here at all.** The step 2 box is dashed because the question raised on #2476 is unanswered: why this is Harper's responsibility rather than CI's or a GTM-coordinated rolling deploy's, and whether forcing rollback of a broken application hinders debugging it. If the answer is that it is not ours, that box comes out and the destination is step 1's staging plus rollback, with no certification at all.

## Why re-plan

#1849 reached **26 files, +6,390/−551 across 108 commits over 5.5 weeks**. Total churn to produce that diff was ~21,200 lines — a **3× rework ratio**. Splitting the cluster-coordination half out into #2301 removed only **16% of the additions and one file** (7,811 → 6,529 insertions), because the bulk was never the peer protocol: it was single-node crash safety, which the split kept. `components/Application.ts` alone went 2,688 → 4,620 lines (+72%), nearly all recovery paths, and tests are 47% of the diff.

The split was along the right axis (verifiability — the unit harness mocks replication, so the peer protocol could not be tested here at all) but the wrong assumption (that the unverifiable half was also the large half). Hence: smaller steps, each with a customer-visible or operator-visible reason to exist.

## What already exists on `main` (do not rebuild)

Worth stating explicitly, because it changes the shape of the later steps:

- `hdb_deployment` rows, `payload_blob`, and whole-operation replication of `deploy_component`.
- **Build-once-replicate-the-artifact, for payload deploys.** The origin already tees `req.payload` through a hash-and-size tap into the row's `payload_blob`, and peers read the tarball back from the replicated row (`components/operations.js`, `sourceExtractionPayload`).
- `get_deployment`, `list_deployments`, `delete_deployment_payload`, payload retention by size and count.

So "one node produces the artifact, every node installs the same bytes" is **already the shipped behaviour for payload deploys**. What is missing is that `package:` and `ref:` deploys resolve independently per node (#2295).

## Progress

| Step | Status | PR |
|---|---|---|
| 1. Build aside, then swap | **merged** | #2345 |
| 2. Isolated-worker validation | **shelved** (2026-09-14) | #2476 (closed), branch `claude/deploy-worker-validation-step2` @ `9734beaab` |
| 3. Root config as an activation effect | not started | — |
| 4. Retention of built asides | **merged** | #2531 |
| 5. `revert_component` | not started | — |
| 6. Deploy from an existing aside | **in progress** | — |
| 7. Build once, replicate the artifact | not started | — |

## The sequence

Each step is a PR unless marked otherwise. Steps are ordered by dependency, and each one is intended to be reviewable in a sitting.

### 1. Build aside, then swap — inside `deploy_component`

**Status: merged** — #2345 (`93752ad79`)

Landed at 13 files, +2,832/−215 — against #1849's 26 files and +6,390, which is the split working as intended. It took 31 cross-model review rounds plus four review passes from @kriszyp; the dominant defect shape by a wide margin was *a rule applied at one site and not its sibling* (a guard, a retire, an ownership read, an attribution invariant). Worth knowing for the steps below, since they touch the same recovery paths.

Build the candidate under a hidden per-deployment directory, run the existing load validation **against the candidate**, then activate it. `main` today does the opposite on both counts: `components/Application.ts:759` renames the *live* tree aside before the candidate exists (so the live path is a construction site for the whole extract + `npm install`), and `components/operations.js:611-653` commits that swap and only *then* validates — so a component that installs cleanly but throws at load goes live anyway while the operation reports an error.

No request/response or progress-phase change. Reuses the aside transaction, the `.in-progress-`/`.retired-` marker contract, and the startup recovery already on `main`.

**Two relaxations, stated because a planning review graded an earlier draft `better-alternative-exists` for leaving them implicit:**

1. **Not strictly gapless.** Activation is two renames, so the live *pathname* is briefly absent. Already-loaded in-memory resources are unaffected, but a component that opens its own files during a request can still observe a gap. This shortens the window from minutes to two renames; it does not remove it. Removing it entirely needs immutable version directories plus an atomically-swapped active locator — see *Possible successors* below.
2. **The ordering fix only bites where validation runs.** `loadValidateComponent` is a no-op on the main thread and the operations API deploys on the main thread, so operator deploys are unvalidated today and remain so after this step. Not a regression, but the guarantee is partial until **step 2**.

### 2. Isolated-worker validation

**Status: SHELVED 2026-09-14.** #2476 is closed; the work is preserved on
`claude/deploy-worker-validation-step2` at `9734beaab` and the branch is not deleted. This is not a pause
with a date on it — the premise is unsettled (see "Why it was shelved" below), and it may not be the right
direction at all.

**Before restarting anything here, read the two subsections at the end of this step.** One decision this
epic recorded as settled turned out to be wrong, and one constraint makes the step as scoped unbuildable.

Run the candidate's load validation in an isolated worker, with the real mount and logical component identity, so it executes regardless of which thread the deploy runs on. Today `validateComponentLoadsExclusive` gates its whole body on `!isMainThread` and the operations API deploys on main, so an operator deploy runs no validation at all — step 1 reordered a no-op there.

**Certification moves to `prepareApplication()`, not `deploy_component`.** A planning review graded the first framing `better-alternative-exists`, correctly: `validateCandidate` is an *optional callback*, and of its four production call sites only `deployComponent()` supplies one — yet `activateCandidateApplication()` writes `.complete`, which step 1's recovery treats as authority for a "build **and validation** complete" candidate. The layer that mints that authority has to require the thing it asserts, or three call sites can still produce `.complete` for an uncertified tree. That is the *one rule, N sites* shape that produced most of step 1's defects.

**What this step does NOT close — tracked to step 3.** A `package:` deploy's root-config entry is written before the build and never rolled back, so: v1 live → v2 written to config → v2 fails validation and is discarded → restart → `installApplications()` prepares v2 and, being certified *at that point*, activates it. The restart path re-validates, so a *broken* v2 still cannot go live — but a v2 the operator was told had failed can. Closing it needs config publication staged with activation, which is step 3. Scope decision taken deliberately: step 2 stays reviewable in a sitting, and the PR will state its guarantee as "within the lifetime of a preparation" rather than as unconditional.

Other decisions from that review, recorded so they are not re-litigated: the validator uses an **unmanaged/ephemeral spawn** that shares option construction with `startWorker` but skips ITC topology, monitoring and restart — a managed thread type would cost O(deploys × workers) message channels and let unrelated broadcasts wait on validators; the transient-validation guard apparatus is **retained**, because a topology-joined validator's `server.registerOperation` announces to main; and isolation is claimed only over JS heap and process-global residue, never over databases, the filesystem or the network.

> **⚠️ The unmanaged-spawn decision above is superseded by evidence — do not restart from it.**
> The cost argument is half wrong: `isEligibleBroadcastRecipient` already excludes a job-type worker
> (`name: THREAD_TYPES.JOB`) from broadcasts, so a job-profile thread never makes unrelated broadcasts wait,
> and the per-peer channel construction is O(workers) for one human-triggered deploy rather than
> O(deploys × workers) in any hot path. What the unmanaged spawn actually cost was **Windows**: a bare
> `new Worker` dies inside its own import graph, before its first statement, exit code 0 and no `error`
> event, so every certification there fails identically (#2494). A thread built by the standard path may or
> may not fix that — the evidence proves the bespoke graph fails, not that the standard bootstrap repairs
> it, and #2494 names the experiment that separates the two.

Split out from step 1 deliberately: it is worker lifecycle and identity plumbing, not filesystem staging, and it is independently reviewable.

#### Why it was shelved

Two reasons, in order of weight.

**1. No available host satisfies both requirements.** A certification load has to be serving-equivalent and
the validator has to be terminable, and nothing has both:

| Host | Serving-equivalent load | Can be force-killed |
|---|---|---|
| A thread in this process | Yes — shares the process's RocksDB handles | **No** under Bun |
| A separate process | **No** — RocksDB's lock is exclusive | Yes |

Under Bun, `terminate()` triggers a NAPI segfault (`manageThreads` and `jobProcess.ts` both avoid it, the
latter draining its loop instead of calling `process.exit`), so the parent can only *ask* a thread to exit —
which a candidate defeats by blocking its event loop or removing the `parentPort` listener. Its concurrency
slot is then held for the life of the process, and two of those stop the node deploying. A helper process is
killable, but cannot open the databases at all: RocksDB's lock is per process, so it fails with
`IO error: While lock file: … Resource temporarily unavailable` the moment `loadRootPlugins` reaches
`getTables()` — and `security/auth.ts` calls `table()` at module scope, so loading fewer plugins does not
avoid it. Opening `readOnly` takes a shared lock and would work, but then any candidate that writes during
load is rejected by certification and fine in production, which is a worse failure than the one prevented.

Note for whoever resumes: a planning review returned `chosen-approach-sound` for the helper-process design,
and that design cannot work. The gate reviews reasoning, not viability. One `fork` would have answered it in
a minute.

**2. The premise itself is contested — and this is the reason it is shelved rather than paused.** Kris asked on #2476 why certification is Harper's responsibility
rather than something done in test environments or CI, why rolling deploy coordinated with GTM is not the
answer if availability of bad deploys is the concern, and — most sharply — whether forcing rollback of a
broken application *hinders* debugging, since a live failure is the best reproduction of itself. That
question is unanswered and is upstream of every implementation choice here.

#### Salvaged, and already landed

Four defects found doing this work were real independently of certification:

- **Fixed and merged**: `package_component` followed `node_modules/harper` into the whole Harper install
(#2487 → #2585, merged as `1496a18ea`).
- **Filed, open**: #2577 (the `harperdb` half of the same packer bug — skipping the path is *not* the fix,
because an archive that omits it cannot restore it on a fresh target) and #2491 (`workerCount` becomes
`undefined` for a non-topology `startWorker`, so `restartWorkers`'s throttle evaluates to `NaN` and a
rolling restart is unthrottled).
- **Closed with the branch**: #2494, the Windows import-graph failure, since that code exists only on the
shelved branch.

The remaining two — the validator leaking process-global RocksDB handles by never calling
`closeLoadedDatabases()`, and a `startWorker` caller with transferred ports crashing the process on restart
via `startCopy` — live in files that exist only on the branch, so they are fixed there and nowhere else.

#### If it resumes

- **Answer the premise question first.** If it does not hold, the honest scope is step 1's staging plus
rollback, with no certification at all.
- **Three open findings are one piece of work**: the bare `Worker` bypassing `collectProvidedWorkerData`
(so `configOverrides` never reaches the validator and its verdict can describe a different environment
than the one serving), the module-local concurrency cap (`prepareApplication` runs on any thread, so N
workers admit 2N validators despite a docstring claiming a process bound), and the shutdown race. All
three resolve by the validator becoming a **managed** thread rather than a freelance one.
- **The bootstrap plumbing for that already exists** in #2476 and is unused by certification:
`extraWorkerData`, `extraTransferList` and `options.noServerStart` on `startWorker`, with reserved-key
and provider-ownership collision checks, plus `terminateProcessGroup`.
- **#2476 also carries four defects worth salvaging independently of certification**, since they are real
today: `package_component` following `node_modules/harper` into the whole install (#2487); a validator
leaking process-global RocksDB handles by never calling `closeLoadedDatabases()`, which blocks an online
`restore_backup`; a `startWorker` caller with transferred ports crashing the process on restart via
`startCopy`; and the certification slot being released rather than handed over. If the step is abandoned
rather than paused, these should be lifted out.

### 3. Root config as an effect of the activation transaction

**Status: not started**

Publish a component's root-config entry as part of the activation, so a rejected or failed deploy cannot leave config naming a release that never went live. Today `operations.js` writes it *before* the build and never rolls it back, so `installApplications()` reinstalls a rejected release at the next restart.

Split out of step 1 after four review rounds: the tree half survived all four, while every one of round 4's six majors was in this area. What that surfaced, and what this step has to solve:

- `env.initSync()` memoizes the config object, so publishing to disk during recovery is not enough — `installApplications()` reads the stale snapshot.
- `atomicWriteFile` does not fsync, so a config write cannot be treated as durable by a transaction that retires the rollback tree on the strength of it.
- The publication lock has to cover every read-modify-write of the document, including `dropComponent`'s `deleteConfigFromFile` — not just deploys.
- Journal entries must be component-scoped, never whole-file snapshots, or two concurrent activations each restore a snapshot predating the other and delete its entry.
- A payload deploy replacing a package-installed component has to explicitly REMOVE the `package` key; "no package" is not "no opinion", or a cold install resolves the old package over the payload release.
- Boot re-installs own no config effect at all, and must journal that fact — otherwise recovery of a crashed boot reinstall deletes the config it was installing from.
- `deleteConfigFromFile` writes to a path rebuilt from `rootPath` rather than the file it parsed, which is a different file on layouts where config does not sit at the root.

### 4. Retention of built asides

**Status: merged** — #2531 (`3ece37294`; docs: HarperFast/documentation#668)

Bound how many unactivated staged builds accumulate, with a config knob. Small and separate on purpose — only meaningful once step 1 leaves asides behind. (`deployment_stagingRetention_maxCount`.)

Landed at 5 files, +843/−26 over 6 commits — the smallest step so far. Two things it established that steps 5 and 6 inherit:

- **A catalog read without the lock is not a verdict.** Dormant builds are catalogued unlocked, off the deploy's critical path, and then every catalogued directory is re-derived under the component's preparation lock before anything is evicted, because a journal can be published into one of them in between. `reconcileDormantBuilds` takes that lock and settles a journal that appeared rather than pruning around it; `pruneDormantBuilds` does the re-derivation and chooses the kept set.
- **`.complete` without a journal is the only shape that is dormant.** `.unsettled` makes a directory residue instead, and only removing the directory clears that marker for workers.

**What it exposed for step 6:** nothing on `main` stages without activating. A successful `activateCandidateApplication` removes `.deploy-staging/` outright, and every failure but a failed compensation discards it — so the dormant builds retention bounds today can only come from a crash between the `.complete` marker and the journal. The integration test plants them rather than producing them (`integrationTests/deploy/staging-retention.test.ts`). Step 6 therefore has to supply the producing half, not just the consuming half.

### 5. `revert_component`

**Status: not started**

A new addressed, idempotent rollback operation, plus the retained-previous tree it reverts to and its own interrupted-revert recovery. Deliberately **not** part of step 1: it is a new public operation *and* new persistent on-disk state. Note this is retention of a *previous live* tree, a different concern from step 2's retention of *unactivated builds* — which is why they are not the same PR.

### 6. Deploy from an existing aside

**Status: not started**

Let `deploy_component` activate a build that is already staged, addressed by deployment id, instead of taking a `package` or `ref`. Needs steps 1 and 4; the row already carries what is required.

### 7. Build once, replicate the artifact — for `package:` and `ref:`

**Status: not started**

Extend the existing payload-blob channel so the origin resolves and packs, and peers install from the replicated blob instead of resolving independently. Closes #2295. Bit-identical bytes across the cluster fall out of this rather than needing a cluster barrier — which is why #2301's stage/activate barrier is **not** being resurrected.

### Not in the sequence

- **#2294** — cross-node activation ordering: two concurrently-originated deploys can still leave nodes on different versions with both reporting success. Build-once does not fix ordering. Stays an issue; it is also the only part that genuinely needs multi-node test infrastructure we do not have.
- **#2273** — the Windows `describe-metadata-upgrade` integration failure seen on #1849 is tracked separately.

### Possible successors (not scheduled)

Both close step 1's pathname gap and are strictly larger than anything above; neither is needed for the sequence to be worth landing.

- **Symlink activation** — the live path becomes a symlink into `.versions//`, so activation is a single atomic rename. Verified on macOS: atomic, `readdir` reports it as a symlink, and a reader holding an open handle to the old target reads it to completion. But `rename(symlink, realDirectory)` fails `EISDIR`, so converting each existing component from a real directory is non-atomic — the window shrinks from *every activation* to *one migration per component, ever*. Windows junction behaviour is unverified. Layers on top of step 1 without redoing it.
- **In-memory routing / module-registry cutover** against a durable active-version manifest — no pathname gap at all, and old loaded resources survive. Needs coordinated changes across worker routing, `EntryHandler`, module resolution, restart semantics and the three startup scans.

## Reference PRs

- **#1849** — the full staged-deploy implementation (staging, activation, recovery, `revert_component`, retention, CLI, DESIGN.md). Reference only; will not merge.
- **#2301** — the cluster coordination protocol (stage/activate barrier, trusted peer operation, separated phases). Reference only; superseded by step 5's design.

Both branches stay available so the individual steps can lift already-reviewed code rather than rewriting it — several rounds of cross-model review and a lot of crash-path test construction are worth reusing.

**They are deliberately kept separate rather than merged into one reference.** #1849's tip deleted `claimStagedDeployment` and `expireOldStagedDeployments` (dead there — #2301 was their only consumer) and makes `activate` / `two_phase` / `deployment_id` `Joi.forbidden()`, while #2301 imports and calls `claimStagedDeployment` and exists to accept those fields. A merged tree would import a function that no longer exists and would mean hand-reconciling a design we've decided not to ship. Each branch is internally coherent on its own; the hybrid would be neither.