HarperFast / HarperFast/harper-pro
Live replication auth-flap on multiple production clusters — Unauthorized(1008) reconnect loop, no monitor fired
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Problem
The 2026-07-06 Datadog audit found a replication auth-flap loop running continuously on production clusters — peers connecting, being rejected `Unauthorized (code: 1008)`, and instantly reconnecting, for weeks. **This is a live, undetected replication incident**: whatever those peers are supposed to replicate is not replicating.
**Affected:** 4 production hosts across 3 customer clusters, combined ~119M log events/7d (~36% of fleet-wide indexed log volume); the single largest host alone accounts for 13% of fleet-wide volume. See the **Customer Impact** field on this issue's project card for specific hostnames/volumes.
Symptom cluster: `Disconnected ... due to "Unauthorized" (code: 1008)` → immediate reconnect; receiving side logs `No node found for IP address ::ffff:, available nodes are , connection will require credentials.` and `did not have valid certificate ... enableRootCAs` warnings; MQTT auth failures with object dumps.
Likely causes to check per cluster: a peer removed from (or never added to) the node registry still configured to connect; certificate/CA misconfiguration after rotation; credential drift. The `No node found for IP` + `Unauthorized` pairing suggests the receiving cluster genuinely doesn't recognize the connecting peer.
## Tasks
- [ ] For each affected host (Customer Impact field): identify the rejecting node + the connecting peer (source IPs are in the `No node found for IP` messages), determine why it's unrecognized, and fix the membership/cert/credential config.
- [ ] Verify replication topology is actually complete afterwards (`cluster_status` all peers connected AND expected peer set matches intended topology).
- [ ] Confirm the log-volume drop in Datadog (expect roughly −36% indexed volume fleet-wide).
Also a direct cost lever: these six patterns are ~119M indexed events/week. The code-side fix (auth backoff + status signal) is filed in harper.
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Research direction
Start with the Customer Impact field for the affected hosts, then inspect the Unauthorized reconnect and No node found for IP log messages in Datadog. Identify the rejecting node, connecting peer, and membership, certificate, or credential cause for each cluster; run cluster_status and verify the intended peers are connected, then confirm the expected Datadog volume drop.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, observability-sre, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100