HarperFast / HarperFast/harper-pro

Adaptive replication routing: dynamic fan-out under egress back-pressure

Open
#218 1 comment 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
area:replication enhancement from-jira
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Overview

Dynamically adjust replication routing topology based on back-pressure from egress network limits, switching to fan-out only when needed.

## Design

Under normal conditions, each node replicates directly to all subscribers (lowest latency). When egress limits are hit:

1. Back-pressure detected in outgoing replication threads → reported to main thread
2. Main thread identifies which outgoing connections are back-pressured
3. Signals a child thread to suspend subscription for the highest-latency remote node
4. Child thread sends subscription suspension request; keeps connection open
5. Remote node redirects that subscriber to another node (re-subscribes indirectly)
6. Main thread monitors: resumes direct subscriptions when back-pressure clears

## Key constraint

Harper doesn't proactively push replication — nodes subscribe to receive it. Back-pressure detection must bridge the gap between the sending thread (which sees network limits) and the main thread (which manages topology).

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.