HarperFast / HarperFast/harper

Missing source blob during a node's full-resync blocks 'ready' forever (EntryHandler #pendingFileReads gate) — wedges deploy_component on peers

Open
#1,472 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

When a node performs a **full-resync / initial scan** (e.g. `add_node`, a new node joining, a re-clone/recovery) and a replicated record references a blob whose **source bytes are permanently unavailable** (deleted / TTL-evicted / incomplete), the node opens a blob read that never completes. That read is only timeout-bounded (`resources/blob.ts` `FILE_READ_TIMEOUT = 60000`), not failed-fast on the *known-permanent* source failure — and `components/EntryHandler.ts` `#checkIfAllComplete()` emits `'ready'` **only when `#pendingFileReads.size === 0`**. So a single stuck read blocks the handler from ever becoming ready: the node stays `lastReceivedStatus:"Waiting"`, the resync never completes, and new rows behind that point don't replicate. Most visibly this fails `deploy_component` — the peer's `awaitDeploymentRow` times out (the harper-pro#402 symptom) because the deployment row is stuck behind the wedge.

The sender *already detects* the permanent failure (`cluster_status.blobReplicationFailures`, #405's ENOENT classification). The gap is that this detection is **never propagated to the receiver's pending read** during initial scan — so the receiver waits out the timeout instead of releasing the read and letting `'ready'` fire.

**Scope (important):** this is **specific to the full-resync / initial-scan path.** Incremental catch-up is **not** affected — the apply loop doesn't block on blobs (the cursor simply lags); verified by a negative control (below). So the trigger is a node doing a full-copy while a referenced source blob is missing — not steady-state operation.

## Mechanism (source)

- `components/EntryHandler.ts` — `#checkIfAllComplete()` emits `'ready'` only when `#isInitialScanComplete && #pendingFileReads.size === 0`. A stuck read keeps the set non-empty (`Initial scan complete, still waiting for N pending file reads`).
- `resources/blob.ts` — the blob read waits for the writer to finish; otherwise a `FILE_READ_TIMEOUT` (60000ms) timer errors `File read timed out … read X bytes, but size is supposed to be Y` / `Blob is incomplete`. No fail-fast path keyed on a known-permanent source failure.
- harper-pro `replication/replicationConnection.ts` — sender records the permanent failure (`blobReplicationFailures`, `markSourceBlobUnavailable`, `isPermanentSourceBlobErrorCode`) but emits no signal that removes the corresponding receiver `#pendingFileReads` entry.

## Evidence (harper-pro 5.1.10; repro in `harper-fabric-lab` `incidents/replicated-deploy-timeout-2026-06-17`)

**RED — full-resync path (`LAG_MODE=wedge`):** receiver logs `Initial scan complete, still waiting for 1 pending file reads`; every peer `system`/`deployprobe` socket `lastReceivedStatus:"Waiting"`; leader `blobReplicationFailures:1`; `deploy_component` peer `Timed out after 120000ms … did not replicate`. Verdict `FIX_INCOMPLETE`.

**GREEN negative control — incremental catch-up (`LAG_MODE=gap`, no `isLeader`):** same source-missing blob, induced via pause→deploy→delete→unpause (the peer resumes from its own resume cursor, no full resync). `NOT_REPRODUCED`, `deploy_ok_all_peers=true`, zero pending file reads — the peer applied the missed row and let the blob lag. This isolates the wedge to the full-resync / initial-scan gate.

## Why the prior fixes don't cover it

#405 advances the *sender's* cursor on ENOENT (sender-side); #451/#453 add a *sender* send timeout (sender-side); #1425 bounds read waits + size-validates (added the 60s bound — still a timeout, not fail-fast); #1334/#1338 surface the peer failure + raise the deploy wait to 120s (symptom); #429 classifies truncated blobs. None makes the **receiver's initial-scan pending blob read fail fast on a sender-detected permanent failure** so `'ready'` can fire.

## Proposed fix (one fix, not another timer)

Propagate the sender's permanent-failure signal (ENOENT/404/incomplete — already detected) to the receiver so it **removes the entry from `EntryHandler.#pendingFileReads` immediately** during initial scan (mark the gap for proactive backfill, consistent with #405's intent), letting `'ready'` fire and the resync complete. Fix the gate, not the timers.

## Reproduce

```sh
# RED (full-resync wedge):
IMAGE=harperfast/harper-pro:5.1.10 LOGGING_LEVEL=debug RESTART_RESET=0 \
bash incidents/replicated-deploy-timeout-2026-06-17/run.sh
docker logs hfl-node2 2>&1 | grep "pending file reads"

# GREEN negative control (incremental — does NOT wedge):
IMAGE=harperfast/harper-pro:5.1.10 LAG_MODE=gap LOGGING_LEVEL=debug RESTART_RESET=0 \
bash incidents/replicated-deploy-timeout-2026-06-17/run.sh
```

## Caveat

Synthetic trigger: the lab *deletes* the source blob. Prod harper-pro#403 was TTL eviction (same gate hangs there during a resync). The `deploy_component` manifestation needs a system-db deployment-payload blob (no TTL) to be missing at the source *while a node full-resyncs* — narrower than steady-state, but real for node joins / re-clones.

---
cc @kriszyp @ldt1996 — surfacing this from the harper-fabric-lab deploy-timeout repro; it's the gap underneath the #402/#403/#405/#429 line. Happy to pair on the `EntryHandler` pending-read release.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with components/EntryHandler.ts and resources/blob.ts, then trace the replication path described for harper-pro replication/replicationConnection.ts. Run incidents/replicated-deploy-timeout-2026-06-17/run.sh in both full-resync and incremental modes, and inspect the pending-file-read and blob-failure logs. Done means a permanently unavailable source blob no longer prevents the initial scan from emitting 'ready', while the incremental negative control remains successful.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.