HarperFast / HarperFast/harper-pro
Replication W10: Performance
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
**Workstream W10 of #430 · performance**
## Summary
Hot-path and structural throughput improvements identified from the protocol read. The incremental items stand alone; the largest wins ride the foundational workstreams (per-origin parallel apply from W4, dedicated threads from W6).
## Pieces
- [ ] **Direct binary relay for the copy path.** Today the base copy *decodes every record only to re-encode it* (`TODO: eventually we don't want to decode, we want to use fast binary transfer`). A passthrough relay would cut copy CPU substantially for large tables and shrink the #421/#266 surface.
- [ ] **Audit-entry field elision in multi-entry transactions** (the `TODO` to elide repeated version/username across entries in one txn) — bandwidth win.
- [ ] **Avoid the per-skipped-record `setImmediate` promise allocation** in the inbound decode loop (frequent on exclusion-filtered tables).
- [ ] **Make `MAX_OUTSTANDING_COMMITS`, `RECEIVE_EVENT_HIGH_WATER_MARK`, `BACK_PRESSURE_INTERVAL` runtime-configurable** (currently hardcoded closure constants) so tuning doesn't require a code change.
- [ ] **Bound the leading-dup fast-skip store read** on the hot path (it does a `getEntry`/`fsPromises.access` per candidate from known-lagging nodes).
- [ ] **`cluster_status` `getSystemInfo` cost** (#260) — shared with W8.
- [ ] **Structural (rides W4/W6):** per-origin parallel apply; dedicated replication threads.
- [ ] **Validation:** WAN latency + high-frequency small-record pattern, 500K records (#300).
## Retires / advances
- [ ] #300 — extend replicationLoad to 500K + WAN + high-freq small-record
- [ ] #421 — large `hdb_analytics` base-copy blocks deploy convergence (copy-path + prioritization angle)
- [ ] #266 — large-clone payload (copy-path angle; transport chunking is in W3)
- [ ] #260 — cluster_status getSystemInfo cost
## Dependencies
Incremental items are independent. Structural wins depend on **W4** and **W6**.
## Effort / risk
**M / low–medium** (incremental); structural wins are sized under W4/W6.
## Acceptance criteria
- Large-table base-copy throughput improves measurably (binary relay).
- The three runtime tunables are adjustable without a rebuild.
- The #300 stress pattern passes within target latency.
---
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Research direction
Start by mapping the replication copy path and inbound decode loop mentioned in the issue, including the binary-transfer TODO, repeated audit fields, and per-skipped-record setImmediate allocation. Review the hardcoded MAX_OUTSTANDING_COMMITS, RECEIVE_EVENT_HIGH_WATER_MARK, and BACK_PRESSURE_INTERVAL, then inspect getEntry/fsPromises.access and cluster_status getSystemInfo. Done means measurable large-table copy improvement, runtime-adjustable tunables, and the #300 WAN stress pattern meeting its latency target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases, distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100