HarperFast / HarperFast/harper-pro
Blob × replication regression tests (Tier 2: randomized chaos/stress harness)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Motivation
Follow-up to harper-pro#411 (deterministic blob×replication regression tests). The deterministic tier locks in known failure modes one fault at a time; this tier hunts the *unknown* ones — the combinatorial interactions of blobs × TTL/eviction × replication/base-copy × concurrent faults that produced the JJill incident and its siblings (HarperFast/harper#1364, HarperFast/harper#1353, harper-pro#409, harper-pro#403). These bugs surfaced under sustained real-world churn, not single deterministic steps, which is exactly what a randomized chaos/stress harness reproduces.
## Scope — randomized chaos/stress harness
Lives in `harper-pro/stressTests/`. Multi-node cluster, run for a fixed duration (or op count) under a continuous, randomized workload + fault injection, then quiesce and run the **invariant oracle from harper-pro#411** (no orphaned references; convergence; bounded orphan files; system/deploy isolation). Reuse that oracle helper — do not reimplement.
### Randomized dimensions
- **Blob size distribution** — incl. the production range (85–420KB) that straddles `min_blob_size`, plus sub-threshold (in-record) and large (multi-MB).
- **TTL** — short enough that records expire *during* the run (drives the eviction/orphan churn).
- **Write rate & mix** — % of writes carrying blobs; insert/update/delete/invalidate ratios; cache vs. non-cache tables.
- **Topology** — node count; leader vs. edge origination.
### Continuous fault injection (randomly timed)
- Fail/drop blob transfers (receive-side write-fail; mid-stream chunk drop; skip-transfer-entirely).
- Delete blob files out from under live records.
- Network partition / WS 1006 drops during active blob transfer.
- SIGKILL a node mid-blob-write (partial file / placeholder header), then restart → forces base-copy.
- Upgrade-style restart to trigger a full copy while orphans exist.
## Requirements
- **Deterministic replay**: seed-driven RNG; log the seed and the full fault timeline so any failing run is reproducible (and can be distilled into a new Tier-1 deterministic test).
- **No silent caps**: log anything bounded/dropped (sampling, top-N) so a clean run isn't mistaken for full coverage.
- **Oracle on quiesce**: stop faults + workload, allow a bounded convergence window, then assert the four invariants on every node. Surface `cluster_status` (`blobReplicationFailures`, resume cursor, backpressure) and blob-store file counts in the failure output.
- **Nightly job** (not per-PR): runs long enough to matter; failures page with seed + timeline.
## Acceptance criteria
- Harness runs a parameterized cluster + randomized blob workload + fault timeline for a configurable duration and reports pass/fail via the shared oracle.
- A failing seed is replayable and reduces to a minimal Tier-1 case.
- Wired into a nightly CI workflow with artifact upload (logs, seed, blob-store snapshots) on failure.
Related: harper-pro#411 (Tier 1, shares the oracle), HarperFast/harper#1364, HarperFast/harper#1353, HarperFast/harper#1369, harper-pro#409, harper-pro#403. Fits the Harper v5 integration test plan (SD Confluence) as the "blob × replication × lifecycle" category.
Contributor guide
Assessment
This issue has not been assessed yet.