HarperFast / HarperFast/harper
deploy_component: verify component loads on post-restart workers
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
In v5.0 and earlier, `deployComponent` performed an inline load of the freshly-deployed component before reporting success — see [the v5.0 path here](https://github.com/HarperFast/harper/blob/v5.0/components/operations.js#L411-L424). That gave the operator confidence: if it loads, it loads; if it fails, the deploy fails with a useful error before the API ever returns 200.
When operations moved to the main thread, that inline-load check could no longer run there (we don't want to ever load component user code on the main thread where deploys are dispatched). What's left in the current handler ([components/operations.js:471-484](https://github.com/HarperFast/harper/pull/760/files)) is a `componentLoader.loadComponent(application.dirPath, pseudoResources, …)` against a `pseudoResources` instance, gated by `!isMainThread && !process.env.HARPER_SAFE_MODE`. That's a worker-local sanity check on whichever HTTP worker happens to be handling the request — useful, but it's not the same as 'the component will load on every worker that will serve traffic after the restart.'
After `manageThreads.restartWorkers('http')` cycles the workers, we never check whether they actually came back up with the component loaded. If the component fails to load post-restart (env-specific missing dep, native binary missing for the runtime, import-time exception), the deploy is already marked `status=success` on the row and the SSE channel is closed. The operator has to discover the failure via traffic errors or by manually inspecting the node.
## Proposal
Add a post-restart readiness rendezvous. The shape I'd suggest, in increasing order of effort:
**Phase 1 — origin self-verifies post-restart**
- After `restartWorkers('http')` returns, origin waits for an 'all workers ready' signal: each freshly-spawned HTTP worker pings the main thread when its `componentLoader` finishes loading every component. Main thread aggregates and resolves once all workers report ready (or any worker reports a load error).
- A per-worker timeout (default ~30s, configurable) bounds the wait so a wedged worker can't hang the deploy reply indefinitely.
- Results land on `hdb_deployment.load_results = { workers: [{worker_id, status, error?}, ...] }`. If any worker fails to load and the project is the one being deployed, the row transitions to `status=failed` with a descriptive `error` — the deploy SSE replays this and Studio surfaces it.
**Phase 2 — peers do the same, results land in peer_results**
- Each peer's `deploy_component` handler waits for its own post-restart workers (same mechanism) before responding to origin's `replicateOperation`. The peer's response carries its `load_results`, which origin records on `peer_results[i].load_results`.
- Origin's aggregate `status=success` requires every peer's `load_results` to be all-green; mixed states surface as `peer_results[i].status='loaded_with_errors'` so the operator can investigate without the whole deploy being marked failed.
**Phase 3 — standalone health op**
- `component_status {project, all_nodes?}` returns the current load state across the cluster (aggregating workers per node and nodes if `all_nodes:true`). Useful as a general health check, not just at deploy time. Could double as the polling target for Studio if the SSE channel was missed.
## Open questions
1. **Worker phone-home plumbing.** `componentLoader.loadComponent` runs as part of normal worker startup, not as a separate verifiable step. Threading a 'ready' signal back to the main thread cleanly requires a new IPC contract — small but real. Worth doing right because once it exists, the same signal helps for the existing 'why is my deploy slow' UX too.
2. **'Loaded' vs 'reachable.'** A worker can have the component loaded but not yet be serving on its port (port-bind race). Phase 1 should probably wait for both: load done *and* listener attached. Already the case for what we consider 'restarted'; we just need to expose it as a signal rather than infer it from process state.
3. **Worker fail semantics.** If one of N HTTP workers fails to load the new component but the others succeed, the cluster keeps serving — the failing worker either retries or stays dead. Should that be `status=success` (cluster healthy) with a warning, or `status=failed`? Plan defaults to: warning on the row's `event_log`, status stays `success` as long as ≥1 worker per node is healthy. Configurable strictness via `deployments.requireAllWorkersLoaded` (default false).
4. **Rollback hook.** Once Slice C (rollback) lands, a failed post-restart verify on a peer is exactly the trigger for an automatic rollback offer. Out of scope for this issue but worth keeping the data model compatible.
## Acceptance
- Single-node: deploy a component that throws at import time. Deploy SSE surfaces a `load-error` event with the import stack; row reaches `status=failed`; CLI exits non-zero.
- Multi-node: deploy a component that loads cleanly on origin but fails on one peer (e.g. simulated import error gated by hostname). Origin's row reaches `status=success` (default lenient mode) with `peer_results[1].status='loaded_with_errors'` carrying the peer's load error; CLI prints the per-peer summary.
- `component_status {project}` returns per-worker / per-node load state, regardless of recent deploy activity.
🤖 Generated by Claude
Contributor guide
Assessment
This issue has not been assessed yet.