HarperFast / HarperFast/harper-pro
Add replication-resilience (restart-under-load) soak test — catch reconnect/convergence regressions (#466/#289 class)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
Add a **replication-resilience (restart-under-load) soak test** to the cluster integration suite. The reconnect-recovery bug harper-pro#466 (a concrete repro of the open #289) was a *latent* defect invisible to every existing test: it only manifests under a specific combination of conditions sustained over time, and our functional cluster tests assert *"a write replicates,"* not *"the cluster re-converges to full mesh after a peer restarts under load."*
## Why existing tests miss this class
Current cluster tests are short-lived, small-data, single-pass. They don't reproduce the conditions #466 needs:
- **Data volume → sustained backpressure.** #466's watchdog backstop is `stop()`-ed under backpressure; small-data tests never build it.
- **Restart a peer *during active replication*** (not a quiet cluster) — the failing TLS-handshake window opens during a peer's secure-context rebuild on restart.
- **Soak / repetition** — the wedge is a *stochastic race* (a connection drop must coincide with a peer's reconnect/TLS-rebuild window); a single pass rarely hits it.
- **Convergence assertion** — tests assert a write arrives, not that *every* `(peer, db)` socket returns to `connected:true` (no stuck pairs).
- Plus: the reconnect-after-restart path is itself unreliable on the macOS dev harness (a self-signed-cert loopback artifact — same `secure TLS connection was established` error), so it must run on Linux/CI.
## Proposed test (new `replication-resilience` category)
A multi-node cluster configured **like production**:
- `enableRootCAs` mesh, **route-less** discovery (peers via `hdb_nodes`, no static routes).
- **Enough data to sustain backpressure** during base copies (a large table + blobs).
- **Restart/kill a peer while replication is actively flowing under load**, repeated several times across different node-pairs.
- **Assert full-mesh re-convergence within a bound:** every node returns to N/N `connected:true`, *no* socket left `connected:false`, within X seconds of each restart. Fail on any stuck pair.
- Soaked/looped to surface the stochastic race; run on **Linux/CI**, not macOS loopback.
This is a "chaos-under-load + assert convergence" test, distinct from the functional write-replication tests. It would have caught #466 (and would catch regressions in the #420/#424/#289/#466 reconnect-recovery family).
## Notes
- Belongs in the v5 integration-test plan / release-testing strategy as a standalone **replication-resilience** category (alongside the functional cluster tests).
- Related: #466, #289, #420, #424, #233, #461 (deploy-resubscribe), #454 (copy-stall — a sibling resilience concern).
- Observability follow-up worth pairing: a `connected:false`-stuck socket is silent (no error/alert) — the same gap that let #466 go unnoticed in the field. A fleet spot-check of per-peer socket counts, and/or surfacing "authed-but-not-replicating" in cluster_status, complements the test.
Contributor guide
Research direction
The issue names no specific file or existing test entry point; start by locating the cluster integration suite and its functional replication tests. Review #466 and the related reconnect issues to understand the failure conditions, then determine how Linux/CI runs integration tests. Done means a standalone replication-resilience test repeatedly restarts peers under load and verifies full-mesh reconvergence within a bounded time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100