Replication receiver buffers blob chunks unbounded when they outrun their record: one worker reached 42.8GB of ArrayBuffers against a documented 5-blob bound, OOM-killing the node repeatedly

## Summary

On the replication receive path, blob chunks that arrive **before their record** are buffered in
memory with backpressure deliberately disabled. The comment justifying this states the exposure is
bounded by the sender's in-flight blob cap — which defaults to **5**. Measured in production, a
single receive worker accumulated **42.8 GB** of `ArrayBuffer`s and the node was OOM-killed three
times in 35 minutes.

At ~223 KB per blob the documented bound is ~1.1 MB. Observed is ~4 orders of magnitude above it.

## The code

`replication/replicationConnection.ts`, inbound `BLOB_CHUNK` handling:

```ts
} else if (!stream.write(blobBody)) {
// The PassThrough's internal queue is over its HWM, meaning the downstream
// file write (via pipeline in saveBlob) can't keep up. Pause the WS until the
// stream drains so blob chunks don't accumulate in memory faster than they
// can be flushed to disk.
if (stream.destroyed || stream.writableEnded) {
// ... skip pausing
} else if (!stream.connectedToBlob) {
// No consumer is attached yet: the blob's chunks have outrun its record, so
// saveBlob — the only thing that drains this PassThrough — has not started
// (receiveBlobs sets connectedToBlob when the record is decoded). Pausing here
// would block the very record that attaches the consumer behind the pause,
// stranding it forever (... the base-copy receive deadlock). Let the
// chunk buffer instead; saveBlob attaches and drains it once the record arrives.
// Exposure is bounded by the sender's in-flight blob cap
// (MAX_OUTSTANDING_BLOBS_BEING_SENT) and the blobsTimer reclaims a truly orphaned stream.
} else {
addPauseReason(); // normal backpressure
...
}
}
```

and the cap that bound relies on:

```ts
const MAX_OUTSTANDING_BLOBS_BEING_SENT = env.get(CONFIG_PARAMS.REPLICATION_BLOBCONCURRENCY) ?? 5;
```

The deadlock-avoidance reasoning is sound — pausing there would strand the record that attaches the
consumer. The problem is only that the fallback is unbounded in practice.

## Evidence

A 4-node cluster, `harper-pro` 5.2.3, page-cache-style table whose records each carry a ~223 KB
blob, with a base copy running. Per-worker `process.memoryUsage()` read over CDP on one receiving
node (16 http workers, MB):

```
PORT WKR heapUsed heapTotal external arrayBuffers
9334 w4 491 791 35,635 35,341 <-- replication receive worker
9330 w0 81 88 127 25
9331 w1 80 83 131 29
9332 w2 79 85 157 27
9333 w3 80 86 254 24
9335 w5 79 86 223 29
... (w6–w15 all ~80 heap / 130–260 external / 25–34 arrayBuffers)

SUM 16 workers: heapUsed 1,697 MB external 38,394 MB arrayBuffers 35,769 MB
```

One worker holds **35.3 GB** of `arrayBuffers`; the other fifteen hold ~30 MB each. `external`
tracks `arrayBuffers` almost exactly, so it is raw byte buffers, not JS objects. That worker is the
one holding the replication connection for the database being copied (it logs the
`Resuming interrupted copy of database …` line for that peer).

Sampled again 6 minutes later on the same worker: `arrayBuffers` 41,371 → 42,819 MB, i.e. still
climbing at ~480 MB/min until the kill.

Three OOM kills on that node, all of `MainThread`:

```
17:49:24 anon-rss 26,220,732 kB total-vm 204,587,416 kB
18:19:18 anon-rss 27,882,268 kB total-vm 189,591,516 kB
```

and a peer receiving a copy from the same source died the same way two minutes before the first:

```
17:47:43 anon-rss 26,083,752 kB total-vm 205,600,536 kB
```

Two independent nodes, near-identical footprints.

## It is not the application workload

The same cluster runs a render pipeline that writes these blob-backed records. Across all 16
workers that accounts for **~1.7 GB of heap** (~80 MB each) and ~30 MB of `arrayBuffers` each. It
is a rounding error against the 42.8 GB in the single receive worker.

Tested directly: pausing the application write queue on the **receiving** node changed nothing (it
OOM'd 11 minutes into the pause, at the same footprint as an unpaused peer). Pausing it on the
**sending** node also changed nothing measurable — the receiver kept growing at ~480 MB/min
throughout. Base-copy volume dominates live write volume roughly 15:1 here.

## Why the stated bound does not hold

Not determined, and worth a maintainer's eye. Two candidates:

1. The cap governs concurrent *sends*, but nothing caps how many **not-yet-connected** PassThroughs
exist on the receiver — a stream whose record never arrives (or arrives much later) keeps its
buffered chunks meanwhile.
2. The same branch deliberately **retains dead streams** in `blobsInFlight` ("deleting it here would
make the next chunk for this fileId recreate a fresh, reader-less PassThrough that backpressures
and re-wedges the WS"), removing them only on a final chunk or via the `blobsTimer` sweep. Under
a copy that skips records — e.g. the `unrecoverable at source … advancing the resume cursor past
it` path — records advance past blobs that will never attach, which is exactly the shape that
leaves orphaned buffered streams awaiting a timeout sweep.

Note (1) and (2) both mean the bound is on the wrong quantity: the sender's *concurrency* does not
bound the receiver's *retention*.

## Impact

The node cannot stay up. Observed cycle: restart with ~25 GB available, consume it in ~4 minutes,
thrash on swap, OOM at ~26–28 GB, repeat roughly every 10–15 minutes. Throughput degrades sharply
as it swaps — measured copy rate fell from ~50 records/sec on fresh memory to 9.8/sec while
thrashing — so the copy takes progressively longer while the node is progressively less able to
serve. The only workaround is to remove the node from load-balancer rotation and let it cycle.

This is not specific to base copy. Any blob-backed replication where chunks can outrun their record
reaches the same branch; a copy simply makes it continuous.

## Asks

1. Should the not-yet-connected case have a bound of its own — a byte ceiling across buffered
PassThroughs, spilling to a temp file or applying backpressure at a level that cannot strand the
attaching record?
2. Should `blobsInFlight` retention be bounded by size as well as by the `blobsTimer` sweep, given
dead and orphaned streams are held deliberately?
3. Is `REPLICATION_BLOBCONCURRENCY` intended to bound receiver memory at all? If so the coupling is
not holding; if not, the comment asserting it should be corrected so the next reader does not
trust it.
4. Any operator-visible signal would help — a `warn` when buffered blob bytes on a connection cross
a threshold. Today the only symptom is an OOM with no preceding replication log line.

## Version

`harper-pro` 5.2.3, `@harperfast/rocksdb-js` 2.7.x. Defaults throughout: `REPLICATION_BLOBCONCURRENCY`
unset (5), `replication_receiveEventHighWaterMark` unset (100), `replication_maxPayload` unset (100 MB).