HarperFast / HarperFast/harper-pro

Replicated component deploys sometimes land on only one node (silent partial deploy)

Open
#163 0 comments 0 reactions 0 assignees View on GitHub
area:replication bug from-jira
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

**Investigative.** `harperdb deploy --replicated=true` of a local component sometimes lands on only one instance in a cluster, leading to inconsistent state (app visible in the UI on one node but not on a sibling, or vice versa). Reported pattern: more likely when the deploy takes a long time, or when there's an error in the component deploy. No reliable repro steps.

## Why this matters

Partial-deploy is silent — the operator sees a successful deploy response from the node they targeted, but sibling nodes don't pick up the component. The drift is only caught later by users hitting an inconsistent endpoint or by manually comparing cluster state.

## To investigate

- Trace the replicated-deploy code path: where does the deploy operation become a replication event, and what guarantees does it carry about cross-node acknowledgement?
- Does a deploy failure on a sibling node (e.g. install error, port collision, missing dep) bubble back to the originating deploy call, or is it swallowed?
- For long deploys: is there a timeout window in the deploy-replication path that drops the replication event without surfacing an error?
- Inventory the failure modes: which combinations of {component already deployed, install error, package extraction error, port-in-use} produce silent partial deploys vs. propagated errors?

## Acceptance criteria

- A replicated deploy either succeeds on all targeted nodes or surfaces a clear error indicating which nodes didn't apply it.
- A small operator-facing way to confirm "is component X present on every node in the cluster" (component-status / cluster-status surface), if not already available.
- Document the expected semantics of `--replicated=true` (best-effort with reporting vs. all-or-nothing).

## Related context

- Slack thread referenced in Jira: [link](https://harperdb.slack.com/archives/C03DLGGN43G/p1771865558152859)

---

Tracked in Jira: [CORE-3037](https://harperdb.atlassian.net/browse/CORE-3037)
**Status:** Not Ready — needs repro / investigation.

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.