HarperFast / HarperFast/harper
Cross-node deploy-transaction ordering: config write + install + replication not serialized across concurrent same-component deploys
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
`deployComponent` (`components/operations.js`) writes the component's YAML config, then extracts/installs, then replicates — but nothing serializes that whole per-component sequence across concurrent deploys of the *same* component. PR #1991 (#1973) added a filesystem lock that closes the `node_modules` corruption window during extract/install, but the wider per-component transaction (config write → install → replicate) can still interleave across two concurrent deploys, leaving disk, config, and peer state disagreeing with each other.
## Repro scenario
1. Deploy A writes `applicationConfig` A via `configUtils.addConfig` (`components/operations.js:416`), then pauses (e.g. in recorder/payload/credential work) before reaching `prepareApplication`.
2. Deploy B writes `applicationConfig` B, acquires the `withComponentPreparationLock` mutex (`components/Application.ts:1070`), and installs B.
3. Deploy A then acquires the lock and installs A.
4. Both calls succeed. Component **A** ends up on disk, but the YAML config on disk is **B**'s. Replication to peers (`server.replication.replicateOperation`, `components/operations.js:613`) then propagates whichever order the two deploys happened to replicate in, which is not guaranteed to match the local disk/config order — peers can end up with a third, different final config/version than the origin.
## Why the existing lock doesn't cover this
`withComponentPreparationLock` in #1991 intentionally scopes serialization to the destructive extract/install filesystem transaction only (the corruption class that caused #1973). It does not cover the config write (which happens before the lock is acquired) or replication (which happens after the lock is released).
Naively widening the lock to span `replicateOperation` was considered and rejected: extending the local filesystem mutex across replication would introduce a distributed deadlock when two nodes originate a same-component deploy concurrently — each origin would hold its local mutex while waiting for the other node to apply the replicated operation.
## What's needed
An explicit cross-node deployment generation/ordering protocol — not a wider local critical section. Candidates to evaluate:
- An ordered commit/version protocol for per-component deploys (e.g. a monotonic deployment generation number that config writes, installs, and replication all key off, with later generations winning and earlier ones detecting/aborting on conflict).
- This may be substantially covered by the `hdb_deployment` tracking work in #641 (replicated deployment records with ordering/rollback) — worth confirming whether that design's generation/ordering story closes this gap, or whether it needs its own explicit treatment.
## Suggested test coverage for a fix
A two-package (or same-package, two concurrent deploy) test that asserts disk contents, on-disk config, and peer-replicated config/version all agree on the same final state after two overlapping deploys.
## Provenance
Surfaced during review of #1991 (in-thread discussion on `components/Application.ts`); confirmed as real and well-characterized by @Ethan-Arrowood's approval on that PR, with a request to track it separately rather than let it evaporate. Deliberately out of scope for #1991, which fixes only the narrower node_modules corruption race from #1973.
Contributor guide
Research direction
Start by reading deployComponent in components/operations.js, withComponentPreparationLock in components/Application.ts, and the replication call around components/operations.js:613. Review issue #641's hdb_deployment design before choosing an ordering approach. Done means overlapping same-component deploys leave disk contents, on-disk config, and peer-replicated config/version consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100