HarperFast / HarperFast/harper
Missing source blob during a node's full-resync blocks 'ready' forever (EntryHandler #pendingFileReads gate) — wedges deploy_component on peers
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
When a node performs a **full-resync / initial scan** (e.g. `add_node`, a new node joining, a re-clone/recovery) and a replicated record references a blob whose **source bytes are permanently unavailable** (deleted / TTL-evicted / incomplete), the node opens a blob read that never completes. That read is only timeout-bounded (`resources/blob.ts` `FILE_READ_TIMEOUT = 60000`), not failed-fast on the *known-permanent* source failure — and `components/EntryHandler.ts` `#checkIfAllComplete()` emits `'ready'` **only when `#pendingFileReads.size === 0`**. So a single stuck read blocks the handler from ever becoming ready: the node stays `lastReceivedStatus:"Waiting"`, the resync never completes, and new rows behind that point don't replicate. Most visibly this fails `deploy_component` — the peer's `awaitDeploymentRow` times out (the harper-pro#402 symptom) because the deployment row is stuck behind the wedge.
The sender *already detects* the permanent failure (`cluster_status.blobReplicationFailures`, #405's ENOENT classification). The gap is that this detection is **never propagated to the receiver's pending read** during initial scan — so the receiver waits out the timeout instead of releasing the read and letting `'ready'` fire.
**Scope (important):** this is **specific to the full-resync / initial-scan path.** Incremental catch-up is **not** affected — the apply loop doesn't block on blobs (the cursor simply lags); verified by a negative control (below). So the trigger is a node doing a full-copy while a referenced source blob is missing — not steady-state operation.
## Mechanism (source)
- `components/EntryHandler.ts` — `#checkIfAllComplete()` emits `'ready'` only when `#isInitialScanComplete && #pendingFileReads.size === 0`. A stuck read keeps the set non-empty (`Initial scan complete, still waiting for N pending file reads`).
- `resources/blob.ts` — the blob read waits for the writer to finish; otherwise a `FILE_READ_TIMEOUT` (60000ms) timer errors `File read timed out … read X bytes, but size is supposed to be Y` / `Blob is incomplete`. No fail-fast path keyed on a known-permanent source failure.
- harper-pro `replication/replicationConnection.ts` — sender records the permanent failure (`blobReplicationFailures`, `markSourceBlobUnavailable`, `isPermanentSourceBlobErrorCode`) but emits no signal that removes the corresponding receiver `#pendingFileReads` entry.
## Evidence (harper-pro 5.1.10; repro in `harper-fabric-lab` `incidents/replicated-deploy-timeout-2026-06-17`)
**RED — full-resync path (`LAG_MODE=wedge`):** receiver logs `Initial scan complete, still waiting for 1 pending file reads`; every peer `system`/`deployprobe` socket `lastReceivedStatus:"Waiting"`; leader `blobReplicationFailures:1`; `deploy_component` peer `Timed out after 120000ms … did not replicate`. Verdict `FIX_INCOMPLETE`.
**GREEN negative control — incremental catch-up (`LAG_MODE=gap`, no `isLeader`):** same source-missing blob, induced via pause→deploy→delete→unpause (the peer resumes from its own resume cursor, no full resync). `NOT_REPRODUCED`, `deploy_ok_all_peers=true`, zero pending file reads — the peer applied the missed row and let the blob lag. This isolates the wedge to the full-resync / initial-scan gate.
## Why the prior fixes don't cover it
#405 advances the *sender's* cursor on ENOENT (sender-side); #451/#453 add a *sender* send timeout (sender-side); #1425 bounds read waits + size-validates (added the 60s bound — still a timeout, not fail-fast); #1334/#1338 surface the peer failure + raise the deploy wait to 120s (symptom); #429 classifies truncated blobs. None makes the **receiver's initial-scan pending blob read fail fast on a sender-detected permanent failure** so `'ready'` can fire.
## Proposed fix (one fix, not another timer)
Propagate the sender's permanent-failure signal (ENOENT/404/incomplete — already detected) to the receiver so it **removes the entry from `EntryHandler.#pendingFileReads` immediately** during initial scan (mark the gap for proactive backfill, consistent with #405's intent), letting `'ready'` fire and the resync complete. Fix the gate, not the timers.
## Reproduce
```sh
# RED (full-resync wedge):
IMAGE=harperfast/harper-pro:5.1.10 LOGGING_LEVEL=debug RESTART_RESET=0 \
bash incidents/replicated-deploy-timeout-2026-06-17/run.sh
docker logs hfl-node2 2>&1 | grep "pending file reads"
# GREEN negative control (incremental — does NOT wedge):
IMAGE=harperfast/harper-pro:5.1.10 LAG_MODE=gap LOGGING_LEVEL=debug RESTART_RESET=0 \
bash incidents/replicated-deploy-timeout-2026-06-17/run.sh
```
## Caveat
Synthetic trigger: the lab *deletes* the source blob. Prod harper-pro#403 was TTL eviction (same gate hangs there during a resync). The `deploy_component` manifestation needs a system-db deployment-payload blob (no TTL) to be missing at the source *while a node full-resyncs* — narrower than steady-state, but real for node joins / re-clones.
---
cc @kriszyp @ldt1996 — surfacing this from the harper-fabric-lab deploy-timeout repro; it's the gap underneath the #402/#403/#405/#429 line. Happy to pair on the `EntryHandler` pending-read release.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with components/EntryHandler.ts and resources/blob.ts, then trace the replication path described for harper-pro replication/replicationConnection.ts. Run incidents/replicated-deploy-timeout-2026-06-17/run.sh in both full-resync and incremental modes, and inspect the pending-file-read and blob-failure logs. Done means a permanently unavailable source blob no longer prevents the initial scan from emitting 'ready', while the incremental negative control remains successful.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100