Blobs as a content layer: location-independent blob identity

## The architectural gap

Harper's record layer already has a real distributed identity model: a record is named by its primary key, versioned, replicated, and convergent — any node can hold it, serve it, and reconcile it. Blob content has none of that. A blob's identity is a node-private counter, which means blob content doesn't exist as a first-class thing in the cluster — it exists only as a side effect of one node's record pipeline. It can't be named across nodes, asked for again, reconciled, moved, or tiered. Every capability we want from blobs — shared storage (#202), branching (#644), resilient replication (#1645), self-healing serving (#2134) — stalls on that same missing property.

The proposal: give blob content its own identity plane, and make placement a separate concern.

- **Identity:** `(node_id, fileId)` — minted exactly once by the node that creates the blob, meaningful on every node, forever. Stored as node-namespaced subtrees (`{db}/{node_id}[.{epoch}]/…`) with today's dense counters and 4096-way fan-out beneath each namespace.
- **Placement:** which volumes, nodes, mounts, or tiers hold a copy — a policy decision, resolved locally at read time, never encoded in the reference (today's `storageIndex` leaves the reference for exactly this reason).

This is the separation storage systems keep converging on — stable names above, movable bytes below — applied with two deliberate deviations from full content-addressing: identity is minted *before* content finishes streaming (so records can reference in-flight blobs, as they do today), and lifecycle stays slaved to record versions (GC unchanged, no refcounting).

## What falls out of stable identity

**Transfer becomes idempotent — push demotes from correctness mechanism to optimization.** Today the inline push inside the record stream is load-bearing: it happens exactly once, from exactly one sender, and when it fails mid-flight we must either wedge (hold the gap: #1337, #1472, harper-pro#683) or lose (advance past it: the harper-pro#403/#405/#429 lineage). The PENDING stubs, receive-in-flight registry, `hasBlobGap`, idle watchdogs, and `sourceBlobUnavailable` taxonomy are all compensations for non-retryability. With stable names, the minting node still pushes proactively (no freshness regression), but a failed push costs nothing: any node pulls `(node_id, fileId)` from any holder, any number of times, materializing via temp-file + atomic rename. The failure machinery isn't fixed — it's obviated.

**Repair becomes convergence.** "The set of blobs node X has minted" is an enumerable, single-writer namespace. Reconciling it — after a partition, a botched copy, a disk swap — is anti-entropy by name, per subtree. Because references stop being rewritten per node, record bytes become identical cluster-wide, making content-hash divergence detection meaningful (harper-pro#432) and giving metadata/blob desync (harper-pro#645) a stable name to reconcile against. The repair sweep (harper-pro#385/#388) becomes an automatic fetch-by-id loop instead of a manual record-mapping exercise.

**Topology changes become metadata operations.** Node dirs are provenance, not ownership: a clone copies them verbatim and mints under its own `node_id` — zero rewrites. Snapshot-restore bumps an epoch so a restored writer can't collide with its own past or a zombie original. A branch (#644) is just another namespace, dissolving the branched-allocator question. Base copy becomes record store + per-node subtree sync.

**Placement becomes a spectrum.** Once content is named independently of location, "where do the bytes live" admits answers the current architecture can't express:

- *Shared storage* (#202): a shared mount is the degenerate case — fetch-from-holder is a filesystem read. Single-writer subtrees are also the friendliest workload a distributed filesystem can be given: no cross-node directory contention, trivial split-brain repair.
- *Sparse replication*: today, replicating a record forces replicating its blob bytes, everywhere. Stable identity permits records-everywhere, content-on-demand — an edge node holds hot blob content only and pulls on miss. For cache-shaped workloads (page bodies, assets) this changes the storage economics of a fleet.
- *Tiering*: `(node_id, fileId)` maps one-to-one onto an object-store keyspace. Cold blob content in S3-compatible storage behind the same fetch-on-miss path is an increment, not a redesign.

**Serving becomes self-healing.** A read miss (#2134) stops being terminal-per-node: the serve path fetches by name from any holder and serves. Blob loss degrades from permanent divergence to cache miss.

## Content flow

The mechanism, end to end:

1. **Write on A** — record commit gated on the blob's durable local write (existing pre-commit gate); the record carries `(A, f1)`.
2. **Record replication A→B** — ref only. B commits immediately and the cursor advances; record replication never stalls on blob bytes again.
3. **Blob arrival at B** — A pushes on commit (keeps today's blob-freshness timing); B pulls if the push hasn't landed by record apply. Both funnel into one single-flight fetch per `(node_id, fileId)`.
4. **Read at B** — local hit serves directly; a miss triggers the same fetch, and the reader tails the file as it materializes (the existing read-while-writing machinery).
5. **Reconciliation sweep** — periodically re-enqueues any ref without a local file. This makes the system convergent without a durable fetch queue: the store itself is the recoverable queue.

```mermaid
sequenceDiagram
participant C as Client
participant A as Node A (origin)
participant B as Node B (replica)

C->>A: PUT record {blob}
A->>A: write blob {db}/A/f1 (durable)
A->>A: commit record with ref (A, f1)
par record stream (existing replication)
A->>B: record + ref (A, f1)
B->>B: commit immediately, cursor advances
and blob push on commit (idempotent prefetch)
A->>B: PUSH blob (A, f1) — pooled mTLS HTTP
B->>B: temp write, atomic rename
end
Note over B: blob absent at record apply?
enqueue PULL (A, f1) — same single-flight fetch,
Range-resumable, retried against any holder
```

The `par` block is the decoupling: the two lanes share no ordering, no backpressure, and no failure coupling. A failed push costs nothing — the pull lane and the sweep converge on the same fetch.

The one deliberate weak point: origin loss before the first copy lands (the window between record commit on A and any peer holding the blob). Push-on-commit keeps that window as small as today's inline push; the pull design does not widen it.

The per-blob resolution lifecycle — states, retry and resume semantics, peer selection — is specified in the companion issue #2148.

## Why now

Three roadmap directions — shared storage (#202), branched databases (#644), and the blob-wedge/loss family (#1645) — are independently converging on the same missing primitive, and the compensation machinery around the third is still growing (harper-pro#336, #659, #683). Meanwhile fleet operations lean ever harder on clone/rehydrate flows that node-local identity makes fragile. Building the primitive once is cheaper than compensating for its absence in three more places.

## Adoption path — each phase independently shippable

1. **Read-both.** Nodes understand the new reference encoding (a new msgpackr ext type — unknown-type reads fail loudly rather than resolving a fileId in the wrong namespace). Inert.
2. **Node-local storage flip.** New blobs mint `(self_node_id, fileId)`; the existing tree grandfathers in place as the node's own namespace — ids unchanged, files unmoved. The wire *protocol* is unchanged — receivers keep re-minting local blobs from the streamed bytes as today — but record sends ship stored value bytes verbatim (`GET_RECORD` responds with the raw value buffer; the typed-structure sync exists to support raw sends), so the new ref encoding does reach peer decoders. That is why this phase is gated on phase 1 being fleet-complete: read-both *is* the compatibility mechanism. Shared storage (#202) is fully enabled at this phase. Downgrade past this point fails loudly, which is the correct failure mode.
3. **Wire v2, per-link.** Send refs; push-as-prefetch; pull-on-miss. Negotiated per link (today's handshake is an exact-match `harperdb-replication-v1` subprotocol, so a small capability mechanism is a prerequisite); v1 links keep today's inline behavior via sender down-conversion. The wedge/lose dilemma disappears link by link.
4. **Retire the compensations; light up placement.** Stubs, gap tracking, and the receive registry become dead code; base-copy-by-rsync, fetch-on-miss serving, sparse replication, and tiering become available as policy.

The absence rule needs no new infrastructure at any phase: blob lifetime is already driven exclusively by record supersession, so *fetch while the current record version references the blob; stop when a newer version drops the ref*. Missing-while-referenced is by definition the loss case, now answered by N holders instead of one.

## Alternatives

- **Snowflake IDs** (#202 thread): the same shape — (machine, unique-within-machine) — with identity hidden in bits. Machine-id assignment remains; snapshot-restore can re-mint from the pre-snapshot clock window (the epoch hazard, implicit instead of explicit); 63 bits force BigInt through hot paths; sparse timestamp ids break low-bits directory sharding (the per-ms sequence bits are ~always zero); existing integer ids can't be absorbed. Node namespaces keep dense counters, short dirs, no clocks, and grandfather the tree in place.
- **Content addressing:** dedupe and integrity for free, but breaks mint-before-write (records reference in-flight blobs), adds GB-scale hashing to the write path, and forces refcounting GC immediately.
- **Record-derived ids:** reverse lookup for sweeps, but coexisting versions and cross-node version ties force a version+tiebreak component — a counter with extra steps. The reverse-lookup win comes cheaper by recording the owning record key in the blob header.
- **Keep patching the push path:** the compensation list *is* the cost curve; each new failure mode buys another stub/registry/watchdog because the transfer stays non-retryable.

## Scope boundaries

Blob lifetime bugs (#1606, #595, #1832, #708, #2062, the retention half of #2134) are GC-policy issues this proposal doesn't change — but their worst case downgrades from permanent divergence to recoverable re-fetch. GC stays record-driven and 1:1; cross-record sharing/dedup becomes *possible* but is out of scope. Prerequisites: the wire-capability mechanism (phase 3) and an epoch hook in the restore path (a namespace lock file fences zombies on shared mounts).

Companion issue: #2148 — the resolution/serving layer (blob store as an internal Resource, owned-relationship model, pluggable sources).

Related: #1645, #202, #644, #1337, #1472, #2134, #141; harper-pro#385, #388, #336, #683, #659, #645, #208, #432.

---

Priority: low — design/direction proposal, no immediate action required. _Drafted with AI assistance (Claude Code)._

Start with companion issue #2148 and the existing record stream, GET_RECORD path, replication handshake, and read-while-writing machinery named here. Map the four adoption phases and their prerequisites before selecting work; done means the selected phase is independently shippable without breaking the stated compatibility and lifecycle boundaries.

Written by the indexing model from the issue text.