Blob replication: receive side is unbounded (send is capped) — a receiver retains every byte for `blobTimeout`, so any base copy OOMs the node at `RX_rate × 900s`

> **Suggested priority: P1.** Re-measured on **5.2.3** (2026-08-19) with a quantitative model and a
> corrected root cause. This is no longer "a stalling peer occasionally spikes one thread": **any
> node receiving a base copy retains every byte it receives for `blobTimeout` (900 s), so its
> footprint is `RX_rate × 900 s`.** At an observed 52 MB/s that is 47 GB against a 34 GB container
> limit, and the node is SIGKILLed every ~11 minutes. Three of four nodes in a cluster are currently
> in this state continuously.

## Summary

Blob replication bounds the **send** side but not the **receive** side.
`MAX_OUTSTANDING_BLOBS_BEING_SENT` (`REPLICATION_BLOBCONCURRENCY`, default 5) gates senders
(`replication/replicationConnection.ts:3073`), but the receiver creates one buffering stream per
incoming blob in an **unbounded** `blobsInFlight` Map. There is **no byte accounting anywhere** in
that path — 25 references to `blobsInFlight`, none of them tracking buffered bytes.

The only thing that ever releases an unconnected stream is the `blobsTimer` idle sweep, on a fixed
`blobTimeout` clock. So the receiver's footprint is a **rate multiplied by a constant**, and there is
no arrival rate at which it stays inside RAM.

## Root cause

`replicationConnection.ts:3786` — when a blob's chunks outrun its record, the receiver deliberately
buffers instead of applying back-pressure. **That decision is correct**: pausing here would block the
very record that attaches the consumer, which is the base-copy receive deadlock the surrounding
guards exist to avoid. The bug is the bound the comment claims:

> `// Exposure is bounded by the sender's in-flight blob cap`
> `// (MAX_OUTSTANDING_BLOBS_BEING_SENT) and the blobsTimer reclaims a truly orphaned stream.`

**Both bounds are false.**

1. **The sender cap does not bound receiver memory.** It caps *concurrent sends* — a blob leaves
`outstandingBlobsBeingSent` once transmitted, not once the receiver connects it to a record. In a
base copy records systematically lag blobs, so the receiver's unconnected set grows without limit
while the sender never exceeds 5 in flight. The two counters measure different things.

2. **The sweep's horizon is a fixed time, not a budget.** A completed-but-unconnected stream is
deliberately left in the Map (`:3762` deletes only `if (stream.connectedToBlob)`), so it is
reachable by the sweep — but only after `blobTimeout` (`:2783`, default 900 000 ms). The sweep
*interval* is already capped at 60 s (`:6382`); the *timeout* is not.

Hence: `steady-state footprint = RX_rate × blobTimeout`.

```
52 MB/s × 900 s = 47 GB required
34 GB container limit
→ OOM at 34 GB / 52 MB/s = 654 s ≈ 11 min (observed: 11 min, 5 SIGKILLs in ~1 h)
```

## What this corrects from the original (2026-08-05) report

The original framing — *a **stalling** peer grows one thread's buffers* — is **refuted**. Stalling is
not required and is in fact protective:

- A node whose copy was **latched** (receiving nothing) sat at **13.3 GB after 121 min uptime**
(~0.11 GB/min). The moment its latch cleared it began accumulating at **1.3 GB/min**.
- Two nodes with **healthy, fast** copies and effectively no blob timeouts accumulated at
**2.4–3.5 GB/min**.

So accumulation tracks *successful receive throughput*, not stalls. The original episode was one way
to reach the condition; a perfectly healthy base copy is the general way.

The original report's "chronic" half (glibc arena residue not returned after the burst) still stands
but is now secondary: on 5.2.3 the acute phase exceeds RAM outright, so nodes are SIGKILLed before
residue matters.

## Evidence — 2026-08-19, harper-pro 5.2.3

4-node Fabric cluster, Linode, identical hardware: 16 core / 31 GB / 34 GB container limit
(`HostConfig.Memory = 36507222016`). Table under copy: `page_cache`, 2,680,322 records, ~68 KB mean
blob. Three nodes receiving a base copy, one not.

**It localizes to a single worker thread per node, and the non-receiving node is exempt:**

| node | receiving a copy? | hot worker | that worker's `arrayBuffers` | node total |
| --- | --- | --- | --- | --- |
| A | yes | `workerIndex` 4 | **35,858 MB** | 36,294 MB |
| B | yes | `workerIndex` 13 | 16,875 MB | 17,364 MB |
| C | yes | `workerIndex` 4 | 12,660 MB | 13,348 MB |
| **D** | **no** | — | 41 MB | **514 MB** |

Node A's `arrayBuffers` (36 GB) exceed its RSS (26 GB) — the remainder is in 17 GB of swap.

**Retention is 1:1 with the network.** Node A, same 30 s window:

| | rate |
| --- | --- |
| network RX (`/proc/net/dev`) | **52 MB/s** |
| `arrayBuffers` growth (24,733 → 26,292 MB) | **52 MB/s** |

Essentially every byte received is retained until the sweep. There is no amplification — and no
release.

**Per-record cost, paired samples over one 2.3-minute window** (records = durable copy-cursor
advance, so this is retention per unit of *banked progress*):

| node | Δ `arrayBuffers` | Δ records | MB retained per record | actual payload |
| --- | --- | --- | --- | --- |
| A | 4,205 MB | 1,207 | **3.48** | ~68 KB |
| C | 3,088 MB | 1,399 | 2.21 | ~68 KB |
| B | 4,300 MB | 2,736 | 1.57 | ~68 KB |

~25–50× the payload per unit of progress, in a table where the mean blob is 68 KB.

Damaged/missing source blobs correlate with the higher end (node A was traversing a region with 8.4%
missing blobs vs node B's 0.8%) but do **not** explain the baseline: node C measured 2.21 MB/record
in a region with **0.0%** missing blobs. The floor is the buffering itself.

## Proposed fixes, by leverage

1. **Split the reclaim timeout (small, surgical, highest leverage).** A stream that is
`writableEnded` but `!connectedToBlob` has already received every chunk; its record either arrives
within seconds or never. It does not need the same deadline as a genuinely stalled in-progress
transfer. Give that class its own short deadline (~30–60 s) and steady state drops 15–30×, to
1.5–3 GB. This is exactly the split #701 already made for `blobGapReconnectMs`, applied to the
reclaim path. Note `blobTimeout` cannot simply be lowered globally — `:401` derives a ping/idle
timeout from `blobTimeout * 2`, and the same value guards legitimately slow transfers.

2. **Add a byte budget to `blobsInFlight` (the correct bound).** Track cumulative buffered bytes and,
past the budget, throttle at the **blob-request** level rather than the socket. Records keep
flowing — which is what drains unconnected streams — so this avoids the deadlock `:3786` is
working around. Memory becomes `O(budget)` instead of `O(rate × time)`.

3. **Spill unconnected chunks to disk.** These bytes are destined for a file anyway (`saveBlob` pipes
to one). Write to the target path on arrival and have `saveBlob` adopt the file when the record
lands. Removes the memory bound entirely.

4. **Fix the ordering (root cause).** The unconnected-buffer path exists only because blobs outrun
records in copy mode. Emit the record before/with its blob and `connectedToBlob` is set on
arrival, so the normal back-pressure path — which works correctly — applies instead.

## Original acute episode — 2026-08-05, harper-pro 5.1.26

Retained because it is the *stalled-peer* variant of the same unbounded receive, and because the
glibc-residue finding is independent.

One node accumulated **9.6 GB of ArrayBuffers on a single thread** — the thread owning that
database's replication socket — while logging **116,441 blob-stream timeouts in 30 minutes** (~65/s,
all distinct blob ids, not retries). `/proc//smaps_rollup` showed the memory is anonymous, not
the LMDB mapping:

```
Rss: 21230092 kB
Anonymous: 21195812 kB <-- essentially all of it
Pss_File: 32892 kB <-- so NOT the LMDB mapping
Swap: 14828564 kB
```

**Aftermath ~1.5 h later, same node** — the ArrayBuffers drained to nothing while RSS+swap did not
move:

| | acute | aftermath |
| --- | --- | --- |
| arrayBuffers (all threads) | ~9.99 GB | **0.38 GB** |
| blob timeouts / hour | ~232k | **0** |
| `RssAnon + VmSwap` | 21.2 + 14.8 = **36.0 GB** | 16.5 + 19.6 = **36.1 GB** |

V8 accounted for ~1.7 GB of the 36 GB; `RssFile` 35 MB; RocksDB `blockCacheUsage` 5.48 GB but
**identical on all four nodes**, so baseline rather than delta. The remainder is arena space glibc
expanded to service the burst and never returned (glibc only trims the main arena, from the top).
Peers did not drift upward with uptime, so this was the episode and not normal aging.

> **Measurement caveat for anyone reproducing this:** instantaneous anon-mapping counts are
> misleading. Caught live on a *healthy* node, mappings sat at 17,651 / 11.12 GB anon for ~60 s, then
> dropped to 5,815 / 8.23 GB within a single 10 s window — 11,836 mappings and 2.89 GB released at
> once (~250 KB each, i.e. above glibc's `MMAP_THRESHOLD`, which `munmap` cleanly). Every node does
> this sawtooth. Sample at least twice ~30 s apart and compare the **minima**.

## How to reproduce / measure

Per-thread `arrayBuffers` is the signal; process RSS is not (it is dominated by swap and arena
residue). Read `process.memoryUsage()` on each worker's inspector port and compare against
`/proc/net/dev` RX over the same window — a healthy receiver should not track RX 1:1.

The issue names replication/replicationConnection.ts:3073, :3786, :3762, :2783, and :6382; start by tracing blobsInFlight, blobsTimer, and blobTimeout through the receive path. Reproduce with per-worker process.memoryUsage() and /proc/net/dev, then define done as a selected fix with bounded receive retention without breaking record attachment or slow transfers.

Written by the indexing model from the issue text.