HarperFast / HarperFast/harper-pro
fix(replication): recovery nets miss a subscription wedged at connected:true / WAITING with no receive progress
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
A replication subscription that parks at `connected: true` + `RECEIVING_STATUS_WAITING` with no receive progress is invisible to **both** reconcile recovery nets, so the wedge is permanent. The stated design intent — "guaranteeing the wedge can never be permanent" ([`subscriptionManager.ts` L119-121](https://github.com/HarperFast/harper-pro/blob/main/replication/subscriptionManager.ts#L119-L121)) — is not met for this shape.
Observed in production: one user database stopped replicating on two receiving nodes simultaneously and stayed dead for **~21 hours** until the containers were restarted manually. Throughout, `cluster_status` reported `connected: true` on every leg, `get_status` reported all components healthy, and the `system` database kept replicating normally on the *same* connections — so every health signal read green.
## Root cause
Both nets exclude this state:
| Net | Gate | Wedged leg |
|---|---|---|
| `findWedgedNodeUrls` | `entry.connected !== true` ([L263](https://github.com/HarperFast/harper-pro/blob/main/replication/subscriptionManager.ts#L263)) | `connected: true` → excluded |
| `findStalledReceivingNodeUrls` → `isReceiveStalled` | `status.status === RECEIVING_STATUS_RECEIVING` ([L347](https://github.com/HarperFast/harper-pro/blob/main/replication/subscriptionManager.ts#L347)) | `WAITING` (0) → excluded |
```js
export function isReceiveStalled(status, now, thresholdMs) {
return (
status != null &&
status.status === RECEIVING_STATUS_RECEIVING && // <- only Receiving
status.lastReceivedTime > 0 &&
now - status.lastReceivedTime >= thresholdMs
);
}
```
`findStalledReceivingNodeUrls` was deliberately scoped to the ping-alive **base-copy** stall (#453), where a follower parks at `Receiving`. A steady-state (non-copy) subscription that stops receiving parks at `WAITING` instead, and nothing covers it.
## Confirmed by elimination
Verified against the live wedged nodes; every other gate was satisfied, so only the status check can have failed:
- `entry.connected` was `true` for the full ~21h → `findWedgedNodeUrls` excluded.
- `isDesired` / `shouldReplicateFromNode` was **true**: `hdb_nodes` self row present with `replicates: true`, peer rows `replicates: true`, `replication.databases: "*"`, local database present. (`selfNodeReplicates` correctly uses `getSync`, so the documented block-cache/Promise-eviction hazard was not in play — though note the `system` database was 6.7 GB here, well inside that danger zone.)
- `lastReceivedTime > 0` (frozen at the failure instant) and age ≫ `RECEIVE_STALL_THRESHOLD_MS` (21h vs 15min) → both satisfied.
- The once-per-episode guard (`receiveStallReconnectAt == null || lastReceivedTime > receiveStallReconnectAt`) was satisfied — the only prior stalled-path kicks on that container were weeks earlier, and `lastReceivedTime` had advanced far past them since.
- **Negative check:** the stalled path's distinct log line (`Reconciling replication subscriptions stalled connected:true with no receive progress`) appeared **zero** times across ~21h (~84 reconcile opportunities at the 15-min cadence), but *did* appear on earlier dates on the same container — so the detector is live code, and `stalledByUrl` was genuinely always empty.
Reproduced independently on two receiving nodes, wedged within ~12 seconds of each other.
## Trigger
The sender crashed and restarted mid-stream. In its final seconds it emitted a record whose typed-structure definition never arrived, and both receivers threw during value decode:
```
[error] [replication]: Error decoding replication message, record id:
typed structures for current decoder[] <- empty
structures for current decoder[[...]]
```
The decode handling itself behaved correctly (logged, skipped, advanced the watermark). The connection then dropped `1006`, reconnected ~30s later for **both** databases — and `system` resumed while the user database never delivered another record.
Note the trigger is generic: **any sender crash under load can produce an undecodable record**, so this is not specific to one deployment.
## Why the obvious fix is wrong
`RECEIVING_STATUS_WAITING` is the **normal** idle/between-batches state — a healthy leg with a quiet source sits there too, with an equally old `lastReceivedTime`. Simply relaxing the status check would force-reconnect every legitimately idle subscription every 15 minutes.
A correct fix needs to distinguish *"idle because the sender has nothing"* from *"wedged while the sender has data"*.
## Constraint on the design
The receiver **cannot** currently make that distinction locally. The shared status buffer carries no advertised sender head — the positions are `CONFIRMATION_STATUS`, `RECEIVED_VERSION`, `RECEIVED_TIME`, `SENDING_TIME`, `LATENCY`, `RECEIVING_STATUS`, `BACK_PRESSURE_RATIO`, `BLOB_FAILURE_COUNT`, `LAST_BLOB_FAILURE_TIME`, `CONNECTION_STATE`, `LAST_LIVENESS_TIME`, `LAST_ERROR_CODE`, `LAST_ERROR_TIME`. `LAST_LIVENESS_TIME` does not help — pongs keep advancing on a wedged-but-open socket exactly as on an idle one.
The **sender** does hold the discriminating signal: during the incident the sender's own `lastCommitConfirmed` for both peers stayed frozen at the failure instant while it held newer commits. That is already tracked per peer per database.
## Candidate directions
1. **Sender-side unconfirmed-commit stall detection** — sender detects "this peer has unconfirmed commits older than threshold" and terminates its side, forcing the receiver to reconnect and resubscribe. Uses data already tracked; the sender is the only party that can tell idle from wedged. Adds a recovery net on the sending side.
2. **Advertise the sender head to the receiver** (new shared-status position / protocol field) so the receiver can compare its applied watermark against the peer's head and keep recovery where `forceReconnect` already lives. Requires a protocol addition.
3. **Targeted recovery at the decode-failure site** — mark the connection for forced reconnect/resubscribe when a value decode throws. Cheap and fixes the observed trigger, but leaves the general blind spot open for any other cause of a `WAITING` park.
Directions 1 and 2 close the gap; 3 only addresses the observed trigger.
## Impact
- Silent: no error after the initial decode failure, and every health surface reads green.
- Unbounded duration: no mechanism recovers it; only a manual restart of the *receiving* node does.
- Correctness-visible to clients: readers on affected nodes served stale data indefinitely while believing they were current.
## Environment
- Reproduced on `harper-pro` 5.1.26; the gap is present in current `main` (verified at `replication/subscriptionManager.ts` L340-351 and L263).
- 3-node mesh, `replication.databases: "*"`, RocksDB storage.
Contributor guide
Assessment
This issue has not been assessed yet.