HarperFast / HarperFast/harper

Base-copy replication wedges in `Receiving` when sender has records referencing blob files missing on disk

Open
#1,337 1 comment 0 reactions 1 assignee Claimed by @ldt1996 View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

When a v5 node attempts a base-copy resync to a peer, and the sender's primary store contains any record whose `Blob` field references a file that doesn't exist on the sender's disk, the receiver's connection wedges permanently in `lastReceivedStatus: "Receiving"`. Base-copy never reaches `COPY_COMPLETE`, the watermark never advances, the blob-bearing database's replication is fully stuck. No errors at the API level; symptom appears only as silent record propagation loss between specific peer-pairs.

## Minimal reproduction

1. Start one v5 node, deploy a fixture with a `Blob` field, insert N records with content above `FILE_STORAGE_THRESHOLD` (8192 bytes) so blobs go to disk files
2. Delete a contiguous block of blob files directly from `/blobs//`
3. Add a fresh v5 peer; receiver requests base-copy resync
4. Observe `cluster_status.connections[0].database_sockets[].lastReceivedStatus` stays at `"Receiving"` indefinitely; only records before the gap apply

```
[blob-gap] source has 60 blob files
[blob-gap] deleted 40 contiguous blob files (simulating v4 gap)
[blob-gap] t=0s receiver status=Receiving record_count=7/60
[blob-gap] t=85s receiver status=Receiving record_count=7/60
```

Migration is NOT required for reproduction — the bug fires on any v5 source whose primary store references a missing blob file. We initially suspected the v4→v5 migration path but ruled it out: migration logs are clean, and the reproduction does not use a v4 binary.

## Test results

| State | Receiver status after ~60s | Record count |
|---|---|---|
| Pre-fix | `Receiving` (stuck) | 7/60 |

Test asserts post-fix behavior. Same setup, same parameters, deterministic.

## Cross-references

- harper-pro#352 — separate transient shared-structure misread surface at boot.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.