Live page_cache records reference deleted blob files: blob lifetime is supersession + 500ms, blind to readers and replication peers (was: #1302 retention half)

> **Status, 2026-08-11.** Root cause settled (maintainer analysis + field verification below):
> blob file lifetime is bound to **record supersession plus a fixed 500 ms timer**, with no
> accounting for outstanding consumers — and on a replicated cluster the dominant consumer that
> falls outside that window is **replication peers**, whose receivers advance past the missing
> blob and commit the record without its bytes (permanent divergence). Reference-aware
> reclamation + automatic backfill is being prototyped by the maintainer. The *user-facing* half
> is mitigated downstream: the prerender plugin (v0.40.0+) reads a cached body to completion
> before committing a response status, so an affected record becomes an ordinary origin proxy
> instead of a truncated `200` (verified: truncated-response counter 62-per-40min → zero). The
> data loss itself continues at the rates below.
>
> **Suggested priority: P1** — behind harper-pro#683, which is the supply line for most of this
> symptom here (detail in comments): its base copies re-stream the whole table, converting every
> concurrent supersession into a diverged record. Fixing #683 shrinks this issue's blast radius;
> the reverse is not true.

## Summary

A 4-node production prerender cluster steadily loses `page_cache` blob files while the records
referencing them stay live. Reads of those records fail with `BlobReadError: Blob file not found`
— the *current* version of a live record points at a path that no longer exists on disk. Present
on 5.1.26 and 5.2.1 alike.

## Root cause (maintainer analysis, field-confirmed)

`RecordEncoder.ts:796` unlinks the prior row's blobs on **every** write, and `deleteBlob`
(`resources/blob.ts:853`) is a bare `setTimeout(unlink, 500)` — no refcount, no read-snapshot
awareness, no replication awareness (the code's own TODO says as much). Two consumer classes fall
outside the 500 ms window:

1. **Replication peers (dominant here).** The origin unlinks a superseded blob while a receiver
is still fetching that fileId; the sender 404s; `isUnrecoverableSourceBlobError` classifies it
permanent; the receiver **advances its resume cursor past the blob and commits the record with
a dangling reference** — permanent until backfilled. Measured on this cluster: the advance-past
log line fired **374,889 times in 6 hours on one node** with a fast inbound base copy
(harper-pro#683 keeps re-triggering those copies), and 25% of sampled serve-path faults were
blobs **still present on a peer** — unambiguous divergence, repairable in principle, except the
repair operation cannot run on a multi-worker instance (harper-pro#684).
2. **In-flight HTTP readers (minority, transient).** `stream()` opens the file by path, lazily,
at stream time; a request whose record-read → file-open gap straddles a concurrent rewrite gets
ENOENT after the 200 is committed. Self-heals on retry.

Full check-by-check field data (existence matrix with positive controls, repeat-fault
distribution, per-node advance-past counts) is in the comments dated 2026-08-11.

## Production evidence

- Continuous fault stream on all four nodes; ~1,000 distinct cache keys fault per 6h cluster-wide
(a **floor** — see measurement caveats below), heaviest on the node with the most accumulated
copy divergence (601 keys/6h) even though its *current* advance-past rate is the lowest.
- Sampled events: 100% `store=page_cache`, 100% file-not-found (`BLOB_GONE_STATUS`), zero
retryable `BLOB_UNAVAILABLE`; blob ids span a 2.59M-id range (not clustered at the counter head).
- Measurement caveats: the pre-mitigation counter only saw blobs failing *mid-stream* (one failing
at `open()` produced no error at all), and Harper analytics under-records vs the logs (~34% low,
measured); treat all counts as floors.

## History: the #1302 hypothesis, tested and retired

This issue originally attributed the loss to the unmerged blob-retention half of #1302
(`kris/blob-invalidate-1302`). A controlled A/B on 5.2.1-equivalent code showed the hunk fixes its
own regression test but converts the loss into an orphan-sweeper deletion, and the maintainer
analysis above showed the invalidate path produces *record-without-reference* (a clean miss), not
*record-with-dangling-reference* (this symptom). That hypothesis is retired; details preserved in
the comment thread.

## Impact

With the downstream mitigation, no more corrupt bytes reach crawlers — the residual cost is lost
cache offload: every affected read becomes an origin fetch plus a re-render. Consumers without the
mitigation still serve truncated/empty bodies as `200`s with every success signal green. On the
replication path, records with dangling refs can also wedge links on 5.2.x (decode-side regression
— see the cross-linked family comment: fc578af re-throws before the cursor advances; deferred
guard harper-pro#432 never landed).

## Asks

1. **Reference-aware reclamation + automatic backfill** (the maintainer's proposed shape): hold a
superseded blob until open read snapshots and peers' resume cursors have passed it; backfill
automatically when a receiver advances past a blob instead of requiring a manual operation.
We can verify a prototype build against production traffic same-day.
2. Until then, harper-pro#684 (make `repair_blob_data` runnable on multi-worker instances) is the
only remediation for accumulated divergence, and it currently cannot run at all.

*(Withdrawn: the original ask for an awaitable blob-readiness API — the downstream read-to-
completion approach covers it, and it only narrows the race anyway.)*

Start by reading RecordEncoder.ts around line 796 and resources/blob.ts around deleteBlob at line 853, then follow the replication receiver's handling of missing blobs and resume cursors. Review the linked field evidence and existing regression coverage before proposing a design. Done means superseded blobs remain available to active readers and peers, and missed blobs trigger automatic backfill without dangling references.

Written by the indexing model from the issue text.