HarperFast / HarperFast/harper-pro

Blob repair sweep has no automatic trigger — a materialized corrupt stub needs an operator to run repair_blob_data

Open
#385 2 comments 0 reactions 0 assignees View on GitHub
area:replication bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

When a blob is replicated between nodes and the transfer is interrupted (read-timeout sweep, WS teardown, or sender error mid-stream), the destination file is left on disk as a corrupt stub **and is never cleaned up or re-requested**. Subsequent reads throw `Blob is incomplete` permanently. The record metadata still references it, so the blob is effectively, silently lost.

## Mechanism

- `replication/replicationConnection.ts` `receiveBlobs` → `saveBlob(localBlob)` runs with the default `deleteOnFailure=false`.
- `core/resources/blob.ts` `writeBlobWithStream` opens the destination with `createWriteStream(filePath, { flags: 'w' })` (truncate/create) and writes the fixed-size header **before** body bytes arrive.
- If the body never completes, the file is left at header-length only. The `pipeline` / `finished(error)` handler does not unlink because `deleteOnFailure` is false.
- A reader hits the header-vs-actual-size mismatch → throws `Blob is incomplete` (`blob.ts` ~line 392). Nothing re-requests the blob from the peer.

## Evidence (prod: `[customer-cluster]`, harper-pro 5.1.0)

- Central node blob store: 753 files, 1.75 MB total, **273 files are exactly 8 bytes**.
- Hexdump of an 8-byte stub: `00 00 00 00 00 1f a7 c5` → type `0` (uncompressed) + declared content size `0x1fa7c5` (~2.07 MB), **zero body bytes**. The header claims ~2 MB; the file is 8 bytes.

## Impact

Silent, permanent blob data loss on any sustained-failing replication link. The record looks intact (size metadata present) but the bytes are unrecoverable from that node. Contributed to ~344 lost document attachments on `[customer-cluster]` (see companion issue).

## Suggested direction

- On interrupted receive, either unlink the partial file or write to a temp name and atomic-rename only on completion — so a failed transfer never leaves a readable-as-corrupt stub.
- Mark the blob for re-request after a failed/timed-out receive.

## Related

- #336 (drop in-flight `blobsInFlight` chunks on failure) and #195 (receive read-timeout / write-after-end race) are adjacent but do not cover the **persisted corrupt stub + no re-request → data loss** path described here.
- Companion: the `[customer-cluster]` durability/observability gap (sustained-failure data loss).

_Found during read-only investigation of the `[customer-cluster]` blob-loss incident, 2026-06-15._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.