HarperFast / HarperFast/harper-pro
Replication W7: Robust sharding (residency-vs-routing split)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
**Workstream W7 of #430 · robust sharding (the residency-vs-routing split)**
## Summary
Sharding is built on "residency" — a node list computed per write and interned to a persistent `residencyId`. The core flaw is that residency is *overloaded*: a transient per-request `replicateTo` is laundered into the **same** persistent `residencyId` as a durable shard policy, so request-scoped intent mutates placement and triggers invalidation broadcasts to nodes that were never targets. Compounding gaps make sharding crash-prone. (Most of this is core `Table.ts` work with replication implications.)
## Root cause / current state
- `replicateTo` becomes the residency list and is interned + stamped on the record (`Table.ts` `getResidency` ~996–1028). An update with `replicateTo:[node2]` makes residency `{node1,node2}`; node3 (previously resident) gets an **invalidate**, and a `replicateFrom:false` read then returns empty — the record looks *deleted* (#212). The stub also re-fetches on a later SQL SELECT (#211).
- `replicateFrom:false` is honored on the main read path (`Table.ts` ~4346) but **bypassed on SQL and blob fetch paths** (#208/#211).
- The cache-load commit **recomputes and re-persists residency** (`Table.ts` ~4896), so a read can mutate placement.
- Residency functions run against **incomplete stubs** (index attrs only) → `Cannot read properties of undefined` (#201). Self-exclusion is enforced at the connection boundary (`replicationConnection.ts:3426`, "Should not connect to self") rather than at routing (#200). Scatter-gather has no merge-aware limit (#198). Constraint errors on relocate/apply aren't fully caught (#199). A "sharded" table silently degrades to replicate-everywhere if `server.shards` isn't populated yet (#257).
## Design direction
1. **First-class distinction between durable residency (shard policy) and per-request targeting**, so `replicateTo` never re-interns a persistent `residencyId` and never broadcasts invalidations to never-target nodes.
2. **One uniform `replicateFrom`/locality choke point** across REST/SQL/blob/source-load — not per-path guards.
3. **Read paths are placement-neutral** — a non-owning read returns local/empty (or fetches without re-persisting residency) and never rewrites ownership.
4. **Residency-function contract** — invoked only with the fields it declares it needs (or always the full record); config-time validation that a sharded table has a live function *and* a populated `server.shards`.
5. **Merge-aware scatter-gather** (limit/offset after the cross-shard merge); exhaustive constraint-error capture on relocate/apply; self-exclusion at residency computation.
## Scope
- [ ] Separate durable residency from per-request `replicateTo` (no persistent re-intern)
- [ ] Single uniform `replicateFrom`/locality choke point (REST/SQL/blob/source-load)
- [ ] Placement-neutral reads (no ownership rewrite on read/cache-load)
- [ ] Residency-function field-presence contract + config-time validation
- [ ] Merge-aware limit/offset for scatter-gather
- [ ] Exhaustive constraint-error capture; self-exclusion at routing
- [ ] Expose shard count (#204) → feeds W8
## Retires / advances
- [ ] #212 — replicateTo update deletes record on non-target node
- [ ] #211 — replicateTo leaks to non-target after SQL SELECT cache-load
- [ ] #208 — X-Replicate-From: none not respected for blob GETs
- [ ] #201 — residency function crash on incomplete stub
- [ ] #200 — "Should not connect to self 1008"
- [ ] #199 — crash after duplicate-value error
- [ ] #198 — `limit` returns too few across shards
- [ ] #257 — records written to all nodes despite sharding config
- [ ] #204 — expose shard count via cluster status
## Dependencies
Largely independent (core-side). Benefits from W8 for the shard-count exposure.
## Effort / risk
**L / medium.** The residency-vs-routing split is the core refactor.
## Acceptance criteria
- A `replicateTo` update neither deletes nor leaks on non-target nodes.
- `X-Replicate-From: none` is honored on blob and SQL paths.
- A sharded write lands on exactly the residency nodes; a SELECT on a non-owner doesn't cache.
- `limit` returns the expected count across shards; sharding misconfig surfaces as an error, not silent full replication.
---
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Research direction
Start in Table.ts at getResidency (~996–1028), the main read path (~4346), and cache-load commit (~4896); compare those paths with SQL, blob, and source-load handling. Review replicationConnection.ts:3426 for the current self-exclusion boundary. Done means durable residency is separate from replicateTo, reads do not rewrite placement, locality is uniform, and the listed acceptance cases pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100