HarperFast / HarperFast/harper-pro

replication: no-backoff subscription-setup retry storm on transient boot-time DNS failure ends in OOM

Open
#327 2 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

If a node restarts while its peer's hostname is briefly unresolvable (`getaddrinfo ENOTFOUND` — routine during container/DNS races at boot), subscription setup enters a **no-backoff retry storm on the main thread** (~1,600 `Setting up subscription with leader for node ` lines/sec), does **not** recover when DNS comes back, stops answering SQL within ~40s, and ends with the container **OOM-killed**.

## Reproduction (internal bench, deterministic)

2-node cluster, 500k-record bulk copy in flight, `docker restart` the receiver. At boot the peer name is unresolvable for ~60s (embedded-DNS race), then resolves fine. Observed on **both** stock `5.1.0-beta.2` and a #321-fix candidate build — pre-existing, independent of #321/#322/#326:

| | stock beta.2 | #322+#326 candidate |
|---|---|---|
| storm lines (~25 min window) | 166,343 | 165,780 |
| SQL responsive after | ~40s, then dead | ~50s, then dead |
| DNS recovered | yes (~60s) | yes (~60s) |
| storm recovered | **no** | **no** |
| end state | unhealthy, `OOMKilled: true` | unhealthy, `OOMKilled: true` |

Timeline (stock): healthy at T+0 → `ENOTFOUND` errors T+0→T+60s → storm T+60s→T+100s (165k+ lines) → log silence/wedge → OOM exit.

## Impact

- Any Fabric instance restarting under transient DNS conditions (multi-tenant hosts, container churn) can spiral to OOM instead of retrying calmly.
- **Blocks mid-copy-restart resilience validation**: the receiver dies before any copy-resume logic (#321, #322, #326) can be exercised — the recovery path those fixes target sits above this.

## Relationship to known issues

Distinct from #323 (worker `subscribe-to-node` racing component load → empty subscription → intentional-close wedge) — same subscription-bootstrap neighborhood, different defect: this one is the main-thread setup loop retrying connection-level failures with no backoff and no recovery.

## Fix surface

Backoff + cap on subscription-setup retries; treat `ENOTFOUND`/connection errors as retryable-with-delay; re-attempt when resolution succeeds; never spin the main thread.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.