HarperFast / HarperFast/harper-pro

Replication silently and permanently drops any transaction whose frame exceeds replication_maxPayload

Open
#711 0 comments 0 reactions 1 assignee Claimed by @ldt1996 View on GitHub
bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

An outgoing replication frame larger than `replication_maxPayload` (default 100 MB) is logged and then silently discarded, with no throw, retry, chunking, or backpressure. The client that wrote the transaction gets a success response, the peer never receives the records, and the loss is permanent. Confirmed on a live two-node v5.2.2 cluster.

## Where

`replication/replicationConnection.ts`, in `sendQueuedData()`: `if (checkExcessMessageSize(frame.position - frame.encodingStart)) return;`. On an oversized frame `checkExcessMessageSize` logs `Message too large to send, size: N` and returns `true`, so the function returns **before** `ws.send()`. The identical predicate on the blob-chunk path throws (`throw new Error('Blob chunk too large')`), but the audit-frame path swallows it. The websocket server is configured with `maxPayload: 10 * 1024 * 1024 * 1024` (`replication/replicator.ts`), 100x the sender's own guard, so the transport is not the constraint.

## Reproduction (live, v5.2.2)

Two-node cluster, insert one transaction whose encoded replication frame exceeds the limit (a bulk insert, or a handful of large records):

- At the real 100 MB default: one `insert` of 1200 records x 100 KB produced a 120,081,888-byte frame. Sender logged `Message too large to send, size: 120081888 remote node: nodeB database: data`. Client `insert` returned `200 {"message":"inserted 1200 of 1200 records"}`. The peer received **0 of 1200** (queried directly, not via logs). Same behavior at a lowered `replication_maxPayload=1000000` with a 2 MB frame.
- **Not a wedged leg:** subsequent smaller transactions replicate normally, the socket stays `connected: true`, `backPressurePercent: 0`.
- **Permanent:** `sendAuditRecord` advances `sentSequenceId = localTime` before `sendQueuedData()`, and the `end_txn` sequence trailer of the *next* transaction advances the receiver's resume cursor past the dropped one. On receiver restart the resume cursor was already at the next transaction's timestamp, so the dropped records are never re-requested.

## Impact

Silent, permanent data loss on replicas for any transaction over the payload limit (bulk loads, large-record writes). The only recovery is a full re-copy (`add_node isLeader:true` / `startTime=0`), which normal operation never triggers.

## Fix

Draft PR converts the silent drop into a loud, recoverable failure (throw so the leg closes and resumes from the un-advanced cursor rather than skipping). The proper long-term fix is mid-transaction chunking (split a large transaction into sub-`maxPayload` frames), which is a larger protocol change tracked as follow-up.

Lavinia, via Claude

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.