HarperFast / HarperFast/harper-pro
Blob × replication regression tests (Tier 1: deterministic cluster integration)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Motivation
A cluster of production incidents on the JJill preprod Fabric cluster (and others) all share one shape that our test suite does not cover: **blobs × TTL/eviction × replication/base-copy × failure-and-recovery**. Each was found in the field, not CI:
- HarperFast/harper#1364 — orphaned blob *reference* created two ways: (1) failed inbound blob transfer + TTL eviction on the origin, (2) expiration-scan force-commit unlinking a blob before the record delete commits.
- HarperFast/harper#1353 — a source-missing blob wedges replication in pre-commit (ENOENT).
- harper-pro#409 / HarperFast/harper#1337 — an orphaned receive-side blob stream crashing the process.
- harper-pro#403 — repeated blob ENOENT permanently pinning the replication resume cursor.
- The JJill incident itself: orphaned blobs → base-copy stream stalls permanently on ENOENT → 100% backpressure → replication wedge → downstream deploy failures.
Today `integrationTests/cluster` has blob fault-injection fixtures (`fixture-blob-fail-injector`, `fixture-blob-fail-transient`, `fixture-large-blob-authoritative`) driving `blobSaveRejectionContainment.test.mjs`. That proves **containment** (a failed inbound save doesn't crash the receiver). It never asserts **convergence** or **no-orphan**. This issue closes that gap with deterministic, per-PR regression tests; harper-pro#412 covers the randomized chaos/stress tier.
## The invariant oracle (shared by both tiers)
After an op sequence + fault injection, **quiesce, then on every node** assert:
1. **No orphaned references** — for every record carrying a blob, the blob is readable and its content-hash equals the source's.
2. **Convergence / liveness** — all nodes reach an identical `(primaryKey → blobHash)` set within a bound; `cluster_status` shows `blobReplicationFailures` stabilized and the resume cursor advancing (never permanently pinned).
3. **Bounded orphan *files*** — leaked blob files (the safe direction of #1364: a skipped unlink) are reclaimed by `cleanupOrphans`; blob-store file count does not grow unbounded across cycles.
4. **Isolation** — `system`/deploy replication stays live while a data-DB blob copy is backpressured/wedged.
A small shared assertion helper implementing (1)–(4) should be the first deliverable; every scenario below ends by calling it.
## Scope — deterministic cluster integration tests (one per failure mode)
Small, seeded, fixed fault points (in the style of the existing `.test.mjs`). Extend the injector as needed beyond receive-side write-fail to also: **skip the blob transfer entirely** (commit metadata, never send the blob), **delete a blob file out from under a live record**, and **drop blob chunks mid-stream**.
- [ ] Shared invariant-oracle helper (items 1–4 above), usable from cluster tests.
- [ ] **Base-copy past a missing blob** (the JJill wedge, highest value): create records whose blobs are missing everywhere, trigger a full base-copy (node restart/join), assert the copy advances past them and the cluster converges — does not stall.
- [ ] **TTL-orphan race** (#1364 Path 1): write short-TTL blob records on an edge node, fail the transfer to the leader, let TTL evict on the edge, trigger a copy → assert convergence + no surviving orphaned reference.
- [ ] **Expiration force-commit** (#1364 Path 2): short TTL + volume sufficient to trip the long-transaction force-commit (`storage.debugLongTransactions: true` to observe) → assert blob files are not unlinked under records that still exist.
- [ ] **Resume cursor not pinned** (#403): inject repeated source-missing-blob ENOENT → assert the cursor advances past (loud) and transient failures hold-then-recover.
- [ ] **Deploy-under-backpressure** (the JJill coupling): wedge/backpressure a data-DB blob copy, then `deploy_component` → assert it still replicates to peers (system-table replication isolated from data-DB blob backpressure).
## Where it lives / reuse
- `harper-pro/integrationTests/cluster/` — multi-node harness + the existing blob fixtures.
- Extend `fixture-blob-fail-injector` with the new injection modes (env-toggled, same pattern).
- Engines: parameterize across RocksDB and LMDB where feasible (the orphan paths differ subtly by engine).
## Acceptance criteria
- Each scenario above is a deterministic test that **fails on the pre-fix code** and passes on current `main` (verify the regression value — e.g. base-copy-past-missing-blob should hang/fail without HarperFast/harper#1353).
- Runs in the normal per-PR integration job (no nightly-only dependency).
- The oracle helper is reused by all scenarios and by the Tier-2 chaos harness.
Related: HarperFast/harper#1364, HarperFast/harper#1353, HarperFast/harper#1369, harper-pro#409, harper-pro#403. Chaos/stress tier: harper-pro#412.
Contributor guide
Assessment
This issue has not been assessed yet.