HarperFast / HarperFast/harper-pro

Inter-node blob fetch timeout surfaces as uncaughtException (verify still occurs after May fixes)

Open
#158 1 comment 0 reactions 0 assignees View on GitHub
area:replication bug duplicate from-jira
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

**Investigative — needs verification on current build.**

When a node attempts to read a blob physically stored on a peer that is temporarily unavailable, the resulting "File read timed out" error has been observed surfacing as an `uncaughtException` in the worker thread rather than being handled gracefully.

## Original customer impact

a customer production cluster upgrade: `[redacted-customer-host]` fetched blobs whose source was on `[redacted-customer-host]`. During central01's compose cycle, MIA01's inter-node blob fetch timed out and threw:

```
[http/2] [error]: uncaughtException Error: Blob error: Error: File read timed out reading from /home/harperdb/hdb/blobs/blobs2/Resilience/9/bda/12b for record ... from [redacted-customer-host]
at _write (node:internal/streams/writable:501:10)
...
at Receiver._write (.../ws/lib/receiver.js:94:10)
```

Errors were transient and self-resolved when central01 came back online — but they surfaced as `uncaughtException` in the meantime.

## Recent related fixes

Two commits since the ticket was filed (2026-03-16) address blob-related `uncaughtException` patterns in `replication/replicationConnection.ts`:

- **22f8777** (2026-05-01) "fix: handle blob save promise rejection to prevent uncaughtException" — adds `.catch()` to the blob save promise so rejections are consumed rather than escaping.
- **a686a55** (2026-05-14) "fix(replication): swallow blob save rejection in outstandingBlobsToFinish" — follow-up: the `.catch()` returned a *new* promise, but the array still held the raw rejected promise. `Promise.all(outstandingBlobsToFinish)` in the `end_txn` onCommit path was still surfacing the rejection — observed in prod as "~35/sec ENOENT spam during catch-up".

Both fix a similar pattern. However, the CORE-3047 stack trace ends in `ws/lib/receiver.js:94`/`_write`, suggesting a *synchronous* throw inside the WebSocket message handler, which is a different mechanism from `Promise.all` rejection. The May fixes may have reduced or eliminated this symptom, but not necessarily covered the exact path.

## To investigate

- Verify with SRE / a customer whether the original "Blob error: ... File read timed out" `uncaughtException` is still occurring on builds ≥ 4.7.26 (after both May fixes shipped).
- The "Blob error:" prefix is constructed at [replication/replicationConnection.ts:797](replication/replicationConnection.ts) where `stream.on('error', () => {})` is intended to swallow:
```ts
stream.on('error', () => {}); // don't treat this as an uncaught error
stream.destroy(new Error('Blob error: ' + error + ...));
```
Confirm whether there is a code path that throws synchronously inside the WS message dispatch before the swallow takes effect.
- Define the desired peer-unavailable behavior (silent retry / queue / propagate to consumer) and document.

## Acceptance criteria (once investigation completes)

- Inter-node blob fetch failures never reach `uncaughtException`.
- Peer-unavailable behavior is defined and documented.
- Repro recipe captured for regression coverage.

---

Tracked in Jira: [CORE-3047](https://harperdb.atlassian.net/browse/CORE-3047) (a customer escalation)
Related: [CORE-3046](https://harperdb.atlassian.net/browse/CORE-3046) (TypeError in incoming replication message handling — likely related)
**Status:** Not Ready — needs SRE verification first.

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.