HarperFast / HarperFast/harper-pro

Bound the bulk-copy walk's lookahead past outstanding blob transfers

Open
#703 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

Follow-up to #699 (whose fixes — copy-cursor banking + in-place blob repair — treat the symptoms; this bounds the disease's blast radius) and a sibling to the in-band per-blob re-request idea from #699's suggestions.

## Proposal

Bound how far the bulk-copy send walk may run ahead of the oldest outstanding blob transfer: after sending a record whose blob is still in flight, allow only a small number of further records (e.g. a few) before pausing the walk until that blob completes. Sender-side only, no wire change (cross-version safe), behind a config key (e.g. `replication.copyBlobLookahead`, `0` = today's unbounded behavior).

## Why

Every deep failure in this family is downstream of the walk running arbitrarily far ahead of unsettled blobs:

- the #426 onCommit-await circular deadlock,
- the consumer-less PassThrough HWM deadlock (copyModeBlobDeadlock),
- unbounded receiver-side unsettled-blob state (the #699 watermark's memory bound exists to contain exactly this),
- large per-cycle re-walk loss on gap reconnects (bounded to the lookahead window instead).

With a small lookahead the durable copy cursor trails the walk head by at most N records, a blob fault costs at most N records per reconnect cycle, and receiver buffering is bounded by construction — a structural guarantee rather than per-bug patches. The stated goal of streaming ahead is keeping the connection saturated, and during a blob-dense copy the blob chunks are the bulk bytes anyway; a record with blobs is usually surrounded by other records with blobs.

## Costs to evaluate (benchmark before defaulting on)

- Head-of-line blocking: one slow blob stalls the walk for its duration, where today other transfers proceed concurrently (bounded by `MAX_OUTSTANDING_BLOBS_BEING_SENT`).
- Mixed tables: a stretch of blob-less records behind one slow blob would stall despite being free to stream.
- Suggested evaluation: blob-dense copy throughput and convergence time at N ∈ {0 (today), 2, 8, 32} on the copyGapCursorBanking harness plus a mixed-table variant.

Origin: Kris's suggestion while reviewing the #699 fix pair (harper-pro#701 / harper#2177).

Contributor guide

Open the contributing guide

Research direction

Start with the copyGapCursorBanking harness and identify the sender-side bulk-copy walk and blob-transfer state it exercises. Evaluate lookahead values N ∈ {0, 2, 8, 32}, including a mixed-table variant, and compare blob-dense throughput and convergence time. Done means the configured bound works without a wire change and the benchmark results support whether a nonzero default is appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.