HarperFast / HarperFast/harper

Two-phase deploy: concurrent activations from different origins can leave the cluster on different versions with both reporting success

Open
#2,294 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Split out of #1849, which adds two-phase (stage → activate) component deploys. Verified still
present in that branch: the activate fan-out runs *after* the origin releases its node-local
component preparation lock (`components/operations.js` — `activateStagedApplication` completes and
releases, then `replicateOperation(buildPhaseOperation('activate', …))`), so nothing establishes an
order between two deploys originated on different nodes.

## The failure

If node A originates deployment D1 and node B originates D2 for the same project at the same time,
each node can apply the other's fan-out last:

- A ends with D2 live
- B ends with D1 live
- **both deployments report success**

The stage barrier does not help. It orders stage-before-activate; it says nothing about
deploy-against-deploy.

The cluster is then silently running two different versions of the same component, and every
observable signal — the operation response, the deployment rows — says both deploys succeeded.
Nothing converges it; it persists until the next deploy of that project happens to land in a
consistent order.

## What a fix requires

A cluster-wide per-project order for activation. Two shapes discussed:

- **Leader serialization** — route activation through a single node per project.
- **Monotonic activation epoch** — persist a per-project counter, compare it under the component
preparation lock during activate, and reject an activation whose epoch is stale.

The epoch option needs a home for the counter that survives restarts. Either is an addition to the
protocol rather than a local fix, which is why it was split out instead of being patched into #1849.

## Notes

- Raised in review of #1849 as that round's only High.
- Exposure today is zero: the two-phase protocol is unshipped. This becomes live the moment #1849
merges, which is why it is milestoned alongside it.
- `ignore_replication_errors` already opts out of the barrier, so any ordering mechanism has to
define its behavior for that path too.

Contributor guide

Open the contributing guide

Research direction

Start with components/operations.js and the activateStagedApplication flow, especially where the preparation lock is released before replicateOperation(buildPhaseOperation('activate', …)). Review the two proposed approaches—leader serialization and a monotonic activation epoch—and define behavior for ignore_replication_errors. Done means concurrent same-project activations establish one cluster-wide order without both deployments reporting success.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.