HarperFast / HarperFast/harper-pro

Add replication-resilience (restart-under-load) soak test — catch reconnect/convergence regressions (#466/#289 class)

Open
#469 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

Add a **replication-resilience (restart-under-load) soak test** to the cluster integration suite. The reconnect-recovery bug harper-pro#466 (a concrete repro of the open #289) was a *latent* defect invisible to every existing test: it only manifests under a specific combination of conditions sustained over time, and our functional cluster tests assert *"a write replicates,"* not *"the cluster re-converges to full mesh after a peer restarts under load."*

## Why existing tests miss this class

Current cluster tests are short-lived, small-data, single-pass. They don't reproduce the conditions #466 needs:
- **Data volume → sustained backpressure.** #466's watchdog backstop is `stop()`-ed under backpressure; small-data tests never build it.
- **Restart a peer *during active replication*** (not a quiet cluster) — the failing TLS-handshake window opens during a peer's secure-context rebuild on restart.
- **Soak / repetition** — the wedge is a *stochastic race* (a connection drop must coincide with a peer's reconnect/TLS-rebuild window); a single pass rarely hits it.
- **Convergence assertion** — tests assert a write arrives, not that *every* `(peer, db)` socket returns to `connected:true` (no stuck pairs).
- Plus: the reconnect-after-restart path is itself unreliable on the macOS dev harness (a self-signed-cert loopback artifact — same `secure TLS connection was established` error), so it must run on Linux/CI.

## Proposed test (new `replication-resilience` category)

A multi-node cluster configured **like production**:
- `enableRootCAs` mesh, **route-less** discovery (peers via `hdb_nodes`, no static routes).
- **Enough data to sustain backpressure** during base copies (a large table + blobs).
- **Restart/kill a peer while replication is actively flowing under load**, repeated several times across different node-pairs.
- **Assert full-mesh re-convergence within a bound:** every node returns to N/N `connected:true`, *no* socket left `connected:false`, within X seconds of each restart. Fail on any stuck pair.
- Soaked/looped to surface the stochastic race; run on **Linux/CI**, not macOS loopback.

This is a "chaos-under-load + assert convergence" test, distinct from the functional write-replication tests. It would have caught #466 (and would catch regressions in the #420/#424/#289/#466 reconnect-recovery family).

## Notes

- Belongs in the v5 integration-test plan / release-testing strategy as a standalone **replication-resilience** category (alongside the functional cluster tests).
- Related: #466, #289, #420, #424, #233, #461 (deploy-resubscribe), #454 (copy-stall — a sibling resilience concern).
- Observability follow-up worth pairing: a `connected:false`-stuck socket is silent (no error/alert) — the same gap that let #466 go unnoticed in the field. A fleet spot-check of per-peer socket counts, and/or surfacing "authed-but-not-replicating" in cluster_status, complements the test.

Contributor guide

Open the contributing guide

Research direction

The issue names no specific file or existing test entry point; start by locating the cluster integration suite and its functional replication tests. Review #466 and the related reconnect issues to understand the failure conditions, then determine how Linux/CI runs integration tests. Done means a standalone replication-resilience test repeatedly restarts peers under load and verifies full-mesh reconvergence within a bounded time.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases, distributed-systems, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.