HarperFast / HarperFast/harper

Blob upload wedges write pipeline when app-supplied uWS request stream stalls mid-upload (no idle watchdog on client-upload path)

Open
#1,862 2 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

### Bug Summary

`writeBlobWithStream` + the commit gate in `startPreCommitBlobsForRecord().complete()` wait forever when an **app-supplied client upload** (uWS HTTP request body) stalls mid-stream under load. The blob's `saving` promise never settles, so the record commit gates on it indefinitely, `outstandingCommit` never clears, and the whole write pipeline wedges — surfacing as a 500 `"Outstanding write transactions have too long of queue, please try again later"` for unrelated writers.

This is the client-upload analog of #1443 (closed) — that fix armed `blobStreamIdleTimeoutMs`/`HARPER_BLOB_STREAM_IDLE_TIMEOUT_MS` on the **replication receive path** only. The app-supplied blob source on the client-upload path has no equivalent idle watchdog.

### Steps to Reproduce

Found while chasing PR #1835 CI failures: a heavy concurrent-writer storm (10 writers × 2MB REPLACEs on one key + 4 tight reader loops) against a **uWS** server. Only the uWS shard fails; the identical assertions pass on the Node HTTP server (~10-12s convergence). Repro isolation:

1. Run `integrationTests/database/qa592-blob-concurrent-writers.test.ts` under `HARPER_UWS_HTTP=1`.
2. A uWS request-body upload stalls mid-stream under the storm; `saveBlob().saving` never settles.
3. Confirmed independent of the durable-unlink-queue work in #1832/#1835 — reproduces identically with that queue disabled (falling back to the old `setTimeout` path).
4. It's a permanent wedge, not a transient slowdown — the next write against the same key fails instantly afterward. Client-side retry/backpressure tolerance does not recover it.

### Expected Behavior

An app-supplied upload stream that stalls should time out (or the commit gate should have a bounded wait) so `outstandingCommit` clears and the write pipeline doesn't wedge for unrelated writers.

### Actual Behavior

No idle watchdog is armed on the client-upload path (only on replication receive, per #1443). A stalled uWS upload permanently wedges all writes until process restart.

### Relevant code

- `resources/blob.ts` — `writeBlobWithStream`
- `startPreCommitBlobsForRecord().complete()` — commit gate
- Compare: replication receive path's `blobStreamIdleTimeoutMs` / `HARPER_BLOB_STREAM_IDLE_TIMEOUT_MS` (armed per #1443)

### Notes

Surfaced by a dev-agent while fixing unrelated CI failures on #1835 (durable blob-unlink queue) — see `harper-1835-fix1` dispatch log/Findings in the `em-state` shepherd queue for the isolation steps. `qa592-blob-concurrent-writers.test.ts` was skipped under uWS on #1835 as a stopgap (assertions are transport-independent and stay covered on the Node HTTP legs); this issue tracks fixing the underlying wedge so uWS coverage can be restored.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.