HarperFast / HarperFast/harper

Replicated redeploy of a static component: new pages 404 on a 5.1.19 cluster until a full cluster restart

Open
#1,935 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

On an internal docs-preview cluster running 5.1.19: redeploying an existing static-component site (`static: { files: 'build/**', urlPath: ..., extensions: ['html'], index: true }`) via `deploy_component` with `restart=true, replicated=true` visibly restarts the receiving node ("Starting replication server" logged as the recycled workers reload the replication component), but newly added pages 404 through the load balancer until the **entire cluster** is restarted. Previously-existing pages keep serving throughout.

**Ruled out** (verified against `harper@5.1.19` from npm and the `v5.1.19` tag):

- Single-node redeploy is clean. The same scenario serves a newly added page immediately — with `restart:true` and even `restart:false` (the deploy:end EntryHandler resume rescan repopulates the static in-memory maps live; `restart:true` additionally recycles http workers via `manageThreads.restartWorkers('http')`, components/operations.js:533).
- Not the #1817/#1820 stale-redeploy class — that behavior change (#1806) ships in 5.2.0-beta.2; 5.1.19 predates it.
- Not #1584 — static + `urlPath` serving verified working on 5.1.19.

So the failure is specific to the replicated leg: one or more peer nodes keep serving their old build behind the LB, and only a full cluster restart converges them.

**Leading candidate:** the peer-side `payload_blob` read stall fixed in 5.1.23 (cd3f50f09, refs #641). On ≤5.1.22 a peer gets a single 20s no-progress window reading the replicated payload and the whole peer deploy fails, leaving that node on the old build. Caveat: a failed peer should surface via the origin's failed-peers error and fail the deploy pipeline, and recent preview deploy runs were green — so either the affected deploys run through a pipeline that doesn't surface the failure, or a different peer-leg mechanism is at work.

**Next steps to confirm:**

- On the next occurrence, request the new page from each node directly (bypassing the LB) to identify the stale node(s); check whether that node's component directory contains the new files, and whether its workers restarted at deploy time.
- Upgrade the cluster to 5.1.23 regardless, to pick up the peer-stall fix; if the symptom persists on 5.1.23, that rules the stall out and the per-node evidence above localizes what's left.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the replicated static-component deploy on 5.1.19 and 5.1.23, then request the new page from each node to locate stale peers. Inspect the deploy:end EntryHandler path and components/operations.js:533, and check peer component files and worker restarts. Done means identifying whether the payload_blob stall is responsible and confirming peer convergence without a full cluster restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.