HarperFast / HarperFast/harper-pro
replication: no-backoff subscription-setup retry storm on transient boot-time DNS failure ends in OOM
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
If a node restarts while its peer's hostname is briefly unresolvable (`getaddrinfo ENOTFOUND` — routine during container/DNS races at boot), subscription setup enters a **no-backoff retry storm on the main thread** (~1,600 `Setting up subscription with leader for node ` lines/sec), does **not** recover when DNS comes back, stops answering SQL within ~40s, and ends with the container **OOM-killed**.
## Reproduction (internal bench, deterministic)
2-node cluster, 500k-record bulk copy in flight, `docker restart` the receiver. At boot the peer name is unresolvable for ~60s (embedded-DNS race), then resolves fine. Observed on **both** stock `5.1.0-beta.2` and a #321-fix candidate build — pre-existing, independent of #321/#322/#326:
| | stock beta.2 | #322+#326 candidate |
|---|---|---|
| storm lines (~25 min window) | 166,343 | 165,780 |
| SQL responsive after | ~40s, then dead | ~50s, then dead |
| DNS recovered | yes (~60s) | yes (~60s) |
| storm recovered | **no** | **no** |
| end state | unhealthy, `OOMKilled: true` | unhealthy, `OOMKilled: true` |
Timeline (stock): healthy at T+0 → `ENOTFOUND` errors T+0→T+60s → storm T+60s→T+100s (165k+ lines) → log silence/wedge → OOM exit.
## Impact
- Any Fabric instance restarting under transient DNS conditions (multi-tenant hosts, container churn) can spiral to OOM instead of retrying calmly.
- **Blocks mid-copy-restart resilience validation**: the receiver dies before any copy-resume logic (#321, #322, #326) can be exercised — the recovery path those fixes target sits above this.
## Relationship to known issues
Distinct from #323 (worker `subscribe-to-node` racing component load → empty subscription → intentional-close wedge) — same subscription-bootstrap neighborhood, different defect: this one is the main-thread setup loop retrying connection-level failures with no backoff and no recovery.
## Fix surface
Backoff + cap on subscription-setup retries; treat `ENOTFOUND`/connection errors as retryable-with-delay; re-attempt when resolution succeeds; never spin the main thread.
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Assessment
This issue has not been assessed yet.