HarperFast / HarperFast/harper
Replicated redeploy of a static component: new pages 404 on a 5.1.19 cluster until a full cluster restart
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
On an internal docs-preview cluster running 5.1.19: redeploying an existing static-component site (`static: { files: 'build/**', urlPath: ..., extensions: ['html'], index: true }`) via `deploy_component` with `restart=true, replicated=true` visibly restarts the receiving node ("Starting replication server" logged as the recycled workers reload the replication component), but newly added pages 404 through the load balancer until the **entire cluster** is restarted. Previously-existing pages keep serving throughout.
**Ruled out** (verified against `harper@5.1.19` from npm and the `v5.1.19` tag):
- Single-node redeploy is clean. The same scenario serves a newly added page immediately — with `restart:true` and even `restart:false` (the deploy:end EntryHandler resume rescan repopulates the static in-memory maps live; `restart:true` additionally recycles http workers via `manageThreads.restartWorkers('http')`, components/operations.js:533).
- Not the #1817/#1820 stale-redeploy class — that behavior change (#1806) ships in 5.2.0-beta.2; 5.1.19 predates it.
- Not #1584 — static + `urlPath` serving verified working on 5.1.19.
So the failure is specific to the replicated leg: one or more peer nodes keep serving their old build behind the LB, and only a full cluster restart converges them.
**Leading candidate:** the peer-side `payload_blob` read stall fixed in 5.1.23 (cd3f50f09, refs #641). On ≤5.1.22 a peer gets a single 20s no-progress window reading the replicated payload and the whole peer deploy fails, leaving that node on the old build. Caveat: a failed peer should surface via the origin's failed-peers error and fail the deploy pipeline, and recent preview deploy runs were green — so either the affected deploys run through a pipeline that doesn't surface the failure, or a different peer-leg mechanism is at work.
**Next steps to confirm:**
- On the next occurrence, request the new page from each node directly (bypassing the LB) to identify the stale node(s); check whether that node's component directory contains the new files, and whether its workers restarted at deploy time.
- Upgrade the cluster to 5.1.23 regardless, to pick up the peer-stall fix; if the symptom persists on 5.1.23, that rules the stall out and the per-node evidence above localizes what's left.
Contributor guide
Research direction
Start by reproducing the replicated static-component deploy on 5.1.19 and 5.1.23, then request the new page from each node to locate stale peers. Inspect the deploy:end EntryHandler path and components/operations.js:533, and check peer component files and worker restarts. Done means identifying whether the payload_blob stall is responsible and confirming peer convergence without a full cluster restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100