HarperFast / HarperFast/harper-pro

Replication W3: Reconnect/storm control & resource bounds

Open
#433 0 comments 0 reactions 0 assignees View on GitHub
area:replication enhancement
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

**Workstream W3 of #430 · Foundation · removes the OOM/CPU-spiral class**

## Status update (2026-07-01)

Partially stale on one detail: subscription-setup retries now have **fixed delays in places** (scheduled `setTimeout` retries in `subscriptionManager.ts`, and a copy-cursor flush-failure backoff landed on the connection side), so "spins with no delay" no longer describes every path. But #327 remains open and the substance stands: there is still no *uniform* backoff discipline (exponential + jitter + cap) across every layer that can initiate work, and #349 (one-shot intent leaking into a persistent listener) and #288 (per-reconnect TLS context churn) are unchanged. Re-verify #327's exact repro against current main when picking this up.

## Summary
Backoff is applied inconsistently across layers, one-shot intents leak into persistent listeners, expensive per-reconnect work amplifies churn into OOM, and oversized payloads are rejected rather than chunked. Together these are the resource-exhaustion bug class under churn.

## Root cause / current state
- The **connection** layer has proper exponential backoff; the **subscription-setup** layer (`subscriptionManager.ts`, on the **main thread**) has only ad-hoc fixed delays — a boot-time DNS failure can still retry-storm (#327, ~1,600 log-lines/s → OOM as originally observed; re-verify current behavior).
- A one-shot `forceResubscribe` intent binds to a *persistent* listener, so every table event re-drives a full resubscribe (#349, ~4,850 lines/s → 100% CPU).
- `tls.createSecureContext` runs **per reconnect** in `monitorNodeCAs`, so reconnect churn allocates unboundedly (#288, heap churn/OOM).
- Oversized messages are dropped above `MAX_PAYLOAD` (`checkExcessMessageSize`) rather than chunked, so a single large record can't traverse the cluster (the live-WS analogue of the abandoned-NATS #266).

## Design direction
1. **One backoff discipline at every layer that can initiate a connection** — subscription-setup gets exponential backoff + jitter + cap, mirroring the connection layer; treat `ENOTFOUND`/connection errors as retryable-with-delay; **never spin the main thread**.
2. **One-shot intents must not bind to persistent listeners** — thread a `fromEvent` flag so persistent re-fires behave as no-ops.
3. **Memoize `createSecureContext`** keyed on CA material so reconnect churn reuses TLS contexts.
4. **Memory-bounded admission control** — back-pressure on bytes/heap outstanding, not just object counts.
5. **Chunk oversized records/payloads** through the streaming (blob-style) transport so replication never depends on a single sub-`MAX_PAYLOAD` message.

## Scope
- [ ] Backoff + jitter + cap on subscription-setup; no main-thread spin
- [ ] `fromEvent` guard so one-shot resubscribe intents don't bind persistent listeners
- [ ] Memoized `createSecureContext` keyed on CA material
- [ ] Bytes/heap-bounded admission control
- [ ] Chunked transport for oversized records/messages

## Retires / advances
- [ ] #327 — no-backoff subscription-setup retry storm on boot DNS failure → OOM
- [ ] #349 — forceResubscribe storm (storm/CPU half)
- [ ] #288 — heap churn/OOM from TLS context churn during reconnect cycles
- [ ] #266 — large-clone oversized payload (reframed onto the WS `MAX_PAYLOAD` path; the NATS-specific issue is stale)

## Dependencies
Complements W1. Independent enough to start immediately (several are localized, high-ROI fixes — good Phase-1 quick wins).

## Effort / risk
**M / low–medium.**

## Acceptance criteria
- A boot-time DNS failure backs off cleanly (no main-thread spin, bounded log volume).
- Reconnect churn reuses TLS contexts (flat heap).
- A record larger than `MAX_PAYLOAD` replicates via chunking.

---
🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start by re-verifying the #327 DNS-failure repro against current main, then inspect subscriptionManager.ts and the connection-side retry paths for existing delay behavior. Trace forceResubscribe listeners, monitorNodeCAs/createSecureContext, checkExcessMessageSize, and the streaming transport. Done means bounded retries and memory use, reused TLS contexts, and successful replication of records larger than MAX_PAYLOAD.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, typescript
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.