HarperFast / HarperFast/harper-pro

Replication W10: Performance

Open
#439 0 comments 0 reactions 0 assignees View on GitHub
area:replication enhancement
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

**Workstream W10 of #430 · performance**

## Summary
Hot-path and structural throughput improvements identified from the protocol read. The incremental items stand alone; the largest wins ride the foundational workstreams (per-origin parallel apply from W4, dedicated threads from W6).

## Pieces
- [ ] **Direct binary relay for the copy path.** Today the base copy *decodes every record only to re-encode it* (`TODO: eventually we don't want to decode, we want to use fast binary transfer`). A passthrough relay would cut copy CPU substantially for large tables and shrink the #421/#266 surface.
- [ ] **Audit-entry field elision in multi-entry transactions** (the `TODO` to elide repeated version/username across entries in one txn) — bandwidth win.
- [ ] **Avoid the per-skipped-record `setImmediate` promise allocation** in the inbound decode loop (frequent on exclusion-filtered tables).
- [ ] **Make `MAX_OUTSTANDING_COMMITS`, `RECEIVE_EVENT_HIGH_WATER_MARK`, `BACK_PRESSURE_INTERVAL` runtime-configurable** (currently hardcoded closure constants) so tuning doesn't require a code change.
- [ ] **Bound the leading-dup fast-skip store read** on the hot path (it does a `getEntry`/`fsPromises.access` per candidate from known-lagging nodes).
- [ ] **`cluster_status` `getSystemInfo` cost** (#260) — shared with W8.
- [ ] **Structural (rides W4/W6):** per-origin parallel apply; dedicated replication threads.
- [ ] **Validation:** WAN latency + high-frequency small-record pattern, 500K records (#300).

## Retires / advances
- [ ] #300 — extend replicationLoad to 500K + WAN + high-freq small-record
- [ ] #421 — large `hdb_analytics` base-copy blocks deploy convergence (copy-path + prioritization angle)
- [ ] #266 — large-clone payload (copy-path angle; transport chunking is in W3)
- [ ] #260 — cluster_status getSystemInfo cost

## Dependencies
Incremental items are independent. Structural wins depend on **W4** and **W6**.

## Effort / risk
**M / low–medium** (incremental); structural wins are sized under W4/W6.

## Acceptance criteria
- Large-table base-copy throughput improves measurably (binary relay).
- The three runtime tunables are adjustable without a rebuild.
- The #300 stress pattern passes within target latency.

---
🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

Start by mapping the replication copy path and inbound decode loop mentioned in the issue, including the binary-transfer TODO, repeated audit fields, and per-skipped-record setImmediate allocation. Review the hardcoded MAX_OUTSTANDING_COMMITS, RECEIVE_EVENT_HIGH_WATER_MARK, and BACK_PRESSURE_INTERVAL, then inspect getEntry/fsPromises.access and cluster_status getSystemInfo. Done means measurable large-table copy improvement, runtime-adjustable tunables, and the #300 WAN stress pattern meeting its latency target.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
databases, distributed-systems, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.