HarperFast / HarperFast/harper

Cross-node deploy-transaction ordering: config write + install + replication not serialized across concurrent same-component deploys

Open
#1,996 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

`deployComponent` (`components/operations.js`) writes the component's YAML config, then extracts/installs, then replicates — but nothing serializes that whole per-component sequence across concurrent deploys of the *same* component. PR #1991 (#1973) added a filesystem lock that closes the `node_modules` corruption window during extract/install, but the wider per-component transaction (config write → install → replicate) can still interleave across two concurrent deploys, leaving disk, config, and peer state disagreeing with each other.

## Repro scenario

1. Deploy A writes `applicationConfig` A via `configUtils.addConfig` (`components/operations.js:416`), then pauses (e.g. in recorder/payload/credential work) before reaching `prepareApplication`.
2. Deploy B writes `applicationConfig` B, acquires the `withComponentPreparationLock` mutex (`components/Application.ts:1070`), and installs B.
3. Deploy A then acquires the lock and installs A.
4. Both calls succeed. Component **A** ends up on disk, but the YAML config on disk is **B**'s. Replication to peers (`server.replication.replicateOperation`, `components/operations.js:613`) then propagates whichever order the two deploys happened to replicate in, which is not guaranteed to match the local disk/config order — peers can end up with a third, different final config/version than the origin.

## Why the existing lock doesn't cover this

`withComponentPreparationLock` in #1991 intentionally scopes serialization to the destructive extract/install filesystem transaction only (the corruption class that caused #1973). It does not cover the config write (which happens before the lock is acquired) or replication (which happens after the lock is released).

Naively widening the lock to span `replicateOperation` was considered and rejected: extending the local filesystem mutex across replication would introduce a distributed deadlock when two nodes originate a same-component deploy concurrently — each origin would hold its local mutex while waiting for the other node to apply the replicated operation.

## What's needed

An explicit cross-node deployment generation/ordering protocol — not a wider local critical section. Candidates to evaluate:
- An ordered commit/version protocol for per-component deploys (e.g. a monotonic deployment generation number that config writes, installs, and replication all key off, with later generations winning and earlier ones detecting/aborting on conflict).
- This may be substantially covered by the `hdb_deployment` tracking work in #641 (replicated deployment records with ordering/rollback) — worth confirming whether that design's generation/ordering story closes this gap, or whether it needs its own explicit treatment.

## Suggested test coverage for a fix

A two-package (or same-package, two concurrent deploy) test that asserts disk contents, on-disk config, and peer-replicated config/version all agree on the same final state after two overlapping deploys.

## Provenance

Surfaced during review of #1991 (in-thread discussion on `components/Application.ts`); confirmed as real and well-characterized by @Ethan-Arrowood's approval on that PR, with a request to track it separately rather than let it evaporate. Deliberately out of scope for #1991, which fixes only the narrower node_modules corruption race from #1973.

Contributor guide

Open the contributing guide

Research direction

Start by reading deployComponent in components/operations.js, withComponentPreparationLock in components/Application.ts, and the replication call around components/operations.js:613. Review issue #641's hdb_deployment design before choosing an ordering approach. Done means overlapping same-component deploys leave disk contents, on-disk config, and peer-replicated config/version consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.