HarperFast / HarperFast/harper-pro
Replication silently and permanently drops any transaction whose frame exceeds replication_maxPayload
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
An outgoing replication frame larger than `replication_maxPayload` (default 100 MB) is logged and then silently discarded, with no throw, retry, chunking, or backpressure. The client that wrote the transaction gets a success response, the peer never receives the records, and the loss is permanent. Confirmed on a live two-node v5.2.2 cluster.
## Where
`replication/replicationConnection.ts`, in `sendQueuedData()`: `if (checkExcessMessageSize(frame.position - frame.encodingStart)) return;`. On an oversized frame `checkExcessMessageSize` logs `Message too large to send, size: N` and returns `true`, so the function returns **before** `ws.send()`. The identical predicate on the blob-chunk path throws (`throw new Error('Blob chunk too large')`), but the audit-frame path swallows it. The websocket server is configured with `maxPayload: 10 * 1024 * 1024 * 1024` (`replication/replicator.ts`), 100x the sender's own guard, so the transport is not the constraint.
## Reproduction (live, v5.2.2)
Two-node cluster, insert one transaction whose encoded replication frame exceeds the limit (a bulk insert, or a handful of large records):
- At the real 100 MB default: one `insert` of 1200 records x 100 KB produced a 120,081,888-byte frame. Sender logged `Message too large to send, size: 120081888 remote node: nodeB database: data`. Client `insert` returned `200 {"message":"inserted 1200 of 1200 records"}`. The peer received **0 of 1200** (queried directly, not via logs). Same behavior at a lowered `replication_maxPayload=1000000` with a 2 MB frame.
- **Not a wedged leg:** subsequent smaller transactions replicate normally, the socket stays `connected: true`, `backPressurePercent: 0`.
- **Permanent:** `sendAuditRecord` advances `sentSequenceId = localTime` before `sendQueuedData()`, and the `end_txn` sequence trailer of the *next* transaction advances the receiver's resume cursor past the dropped one. On receiver restart the resume cursor was already at the next transaction's timestamp, so the dropped records are never re-requested.
## Impact
Silent, permanent data loss on replicas for any transaction over the payload limit (bulk loads, large-record writes). The only recovery is a full re-copy (`add_node isLeader:true` / `startTime=0`), which normal operation never triggers.
## Fix
Draft PR converts the silent drop into a loud, recoverable failure (throw so the leg closes and resumes from the un-advanced cursor rather than skipping). The proper long-term fix is mid-transaction chunking (split a large transaction into sub-`maxPayload` frames), which is a larger protocol change tracked as follow-up.
Lavinia, via Claude
Contributor guide
Assessment
This issue has not been assessed yet.