HarperFast / HarperFast/harper

Replication: no backoff or status signal on persistent auth failure — reconnect loop generated 73M log events/week, undetected

Open
#1,635 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Problem

A Datadog audit (7 days, 2026-06-30 → 2026-07-06) found that **~36% of all indexed log volume** is a single replication auth-failure reconnect loop:

| message pattern | 7d count |
|---|---|
| `Disconnected from wss://:9933 (db: ""), due to "Unauthorized" (code: 1008)` | 34.2M |
| `Connected to wss://:9933, db: ` | 27.5M |
| `Incoming client connection from ::ffff: did not have valid certificate, you may need turn on enableRootCAs...` | 16.8M |
| `No node found for IP address ::ffff:, available nodes are , connection will require credentials.` | 16.8M |
| `No authorization provided` | 11.4M |
| `MQTT auth failed: no credentials / not authorized {` + multi-line object dump | 12.1M |

That's ~73M log events in a week from peers that connect, get rejected as Unauthorized, and immediately reconnect — continuously, for weeks. **No monitor fired and nothing surfaced in `get_status`/`cluster_status`** — the flap was only discovered by auditing Datadog billing data.

Three defects compound here:

1. **No auth-aware backoff.** A persistent auth/authorization rejection (WS close 1008, missing credentials, unknown-node) is retried on the same tight schedule as a transient network error. Auth failures don't self-heal in milliseconds; retrying instantly just burns CPU, connections, and log volume.
2. **Every cycle logs 3–6 lines at warn/error**, including the full node list in the `No node found for IP` message (unbounded message size on large clusters) — with no dedup (harper-pro#262).
3. **No durable status signal.** A peer in a persistent auth-failure loop should be a single `error`-level status entry (per peer, with occurrence count and last-error detail), visible in `get_status`, not millions of identical log lines.

## Proposed fix

- Classify connection failures: auth-class rejections (1008 Unauthorized, `No authorization provided`, unknown node/cert rejection) get exponential backoff with a high cap (e.g. 30s → 10min), distinct from transient network errors.
- One status entry per peer (e.g. `replication.connection.`), set on failure with the close code/reason, cleared on successful authenticated connect — so the flap is visible and alertable via the status system (PR #372 direction).
- Log the first occurrence and every Nth (or once per backoff step) with a repeat count; drop the full node list from the hot-path message (log it once at debug).
- Surface the last connection error per socket in `cluster_status` (harper-pro#214 / #437 Tier 1).

## Related
harper-pro#437 (Replication W8 observability), harper-pro#262 (log dedup), harper-pro#214 (last connection error), PR #372 (logger.status bridge). Live flapping clusters tracked separately in harper-pro.

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start by reading the replication connection handling and the status-system entry points named in the issue, especially get_status and cluster_status, along with the PR #372 direction. Trace how auth failures, reconnects, and logs are currently represented. Done means auth failures back off, repeated logs are reduced, and per-peer errors with counts and last details are visible and clear after a successful authenticated connection.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, distributed-systems, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.