HarperFast / HarperFast/harper-pro

Replication W7: Robust sharding (residency-vs-routing split)

Open
#436 0 comments 0 reactions 0 assignees View on GitHub
area:clustering area:replication enhancement
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

**Workstream W7 of #430 · robust sharding (the residency-vs-routing split)**

## Summary
Sharding is built on "residency" — a node list computed per write and interned to a persistent `residencyId`. The core flaw is that residency is *overloaded*: a transient per-request `replicateTo` is laundered into the **same** persistent `residencyId` as a durable shard policy, so request-scoped intent mutates placement and triggers invalidation broadcasts to nodes that were never targets. Compounding gaps make sharding crash-prone. (Most of this is core `Table.ts` work with replication implications.)

## Root cause / current state
- `replicateTo` becomes the residency list and is interned + stamped on the record (`Table.ts` `getResidency` ~996–1028). An update with `replicateTo:[node2]` makes residency `{node1,node2}`; node3 (previously resident) gets an **invalidate**, and a `replicateFrom:false` read then returns empty — the record looks *deleted* (#212). The stub also re-fetches on a later SQL SELECT (#211).
- `replicateFrom:false` is honored on the main read path (`Table.ts` ~4346) but **bypassed on SQL and blob fetch paths** (#208/#211).
- The cache-load commit **recomputes and re-persists residency** (`Table.ts` ~4896), so a read can mutate placement.
- Residency functions run against **incomplete stubs** (index attrs only) → `Cannot read properties of undefined` (#201). Self-exclusion is enforced at the connection boundary (`replicationConnection.ts:3426`, "Should not connect to self") rather than at routing (#200). Scatter-gather has no merge-aware limit (#198). Constraint errors on relocate/apply aren't fully caught (#199). A "sharded" table silently degrades to replicate-everywhere if `server.shards` isn't populated yet (#257).

## Design direction
1. **First-class distinction between durable residency (shard policy) and per-request targeting**, so `replicateTo` never re-interns a persistent `residencyId` and never broadcasts invalidations to never-target nodes.
2. **One uniform `replicateFrom`/locality choke point** across REST/SQL/blob/source-load — not per-path guards.
3. **Read paths are placement-neutral** — a non-owning read returns local/empty (or fetches without re-persisting residency) and never rewrites ownership.
4. **Residency-function contract** — invoked only with the fields it declares it needs (or always the full record); config-time validation that a sharded table has a live function *and* a populated `server.shards`.
5. **Merge-aware scatter-gather** (limit/offset after the cross-shard merge); exhaustive constraint-error capture on relocate/apply; self-exclusion at residency computation.

## Scope
- [ ] Separate durable residency from per-request `replicateTo` (no persistent re-intern)
- [ ] Single uniform `replicateFrom`/locality choke point (REST/SQL/blob/source-load)
- [ ] Placement-neutral reads (no ownership rewrite on read/cache-load)
- [ ] Residency-function field-presence contract + config-time validation
- [ ] Merge-aware limit/offset for scatter-gather
- [ ] Exhaustive constraint-error capture; self-exclusion at routing
- [ ] Expose shard count (#204) → feeds W8

## Retires / advances
- [ ] #212 — replicateTo update deletes record on non-target node
- [ ] #211 — replicateTo leaks to non-target after SQL SELECT cache-load
- [ ] #208 — X-Replicate-From: none not respected for blob GETs
- [ ] #201 — residency function crash on incomplete stub
- [ ] #200 — "Should not connect to self 1008"
- [ ] #199 — crash after duplicate-value error
- [ ] #198 — `limit` returns too few across shards
- [ ] #257 — records written to all nodes despite sharding config
- [ ] #204 — expose shard count via cluster status

## Dependencies
Largely independent (core-side). Benefits from W8 for the shard-count exposure.

## Effort / risk
**L / medium.** The residency-vs-routing split is the core refactor.

## Acceptance criteria
- A `replicateTo` update neither deletes nor leaks on non-target nodes.
- `X-Replicate-From: none` is honored on blob and SQL paths.
- A sharded write lands on exactly the residency nodes; a SELECT on a non-owner doesn't cache.
- `limit` returns the expected count across shards; sharding misconfig surfaces as an error, not silent full replication.

---
🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start in Table.ts at getResidency (~996–1028), the main read path (~4346), and cache-load commit (~4896); compare those paths with SQL, blob, and source-load handling. Review replicationConnection.ts:3426 for the current self-exclusion boundary. Done means durable residency is separate from replicateTo, reads do not rewrite placement, locality is uniform, and the listed acceptance cases pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
databases, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.