HarperFast / HarperFast/harper
Consensus-replicated lane for workflow state
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
Part of #752 (Durable Execution v0.1 epic).
## Problem
Harper's current replication is asynchronous, peer-to-peer, last-writer-wins on timestamp with node-name tiebreak — verified in [\`Table.ts\`](https://github.com/HarperFast/harper/blob/main/resources/Table.ts) at \`precedesExistingVersion\` (around line 4089) and supported by the node-name → node-id mapping in \`nodeIdMapping.ts\`. There is no Raft, no quorum, no leader election in steady state: \`replicateTo\` controls residency and an optional per-request commit-confirmation count, not a quorum write.
This is fine for application data, but unacceptable for workflow execution. A failure mode: node A commits a \`step-7-completed\` checkpoint locally, then crashes before the change propagates. Node B picks up the workflow, doesn't see the checkpoint, re-executes step 7 — and re-charges Stripe. LWW conflict resolution on recovery does not save us; the checkpoint can be lost outright.
## Proposed shape
A **per-shard Raft group on a dedicated workflow-state subsystem, alongside the existing replication path** — not replacing it. Existing data continues to use the current high-throughput peer-to-peer lane; workflow checkpoints and step outputs use a quorum-replicated commit log.
- **Sharding key**: workflow ID. Each workflow has a strict serial order; cross-workflow operations remain eventually consistent.
- **Quorum**: standard Raft majority. A workflow step is not acknowledged committed until a quorum has it.
- **Co-location**: the shard for workflow W lives on the same nodes (or a subset) that hold W's checkpoint data, so the workflow runtime can wake on whichever node holds the leader of W's shard without an extra hop.
- **Leader election + shard placement**: out-of-band; placement should be observable and rebalanceable as load metrics develop.
## Why this is not satisfied by #483
#483 (exclusive record locks) is concurrency control — it prevents two nodes writing the same record at once by broadcasting a lock acquisition. It does **not** make a locally-committed write durable across node failure. The two are complementary: locking ensures a single executor; consensus ensures the executor's checkpoints are durable before the step is acknowledged.
## Open design questions
- Do we use an in-house Raft implementation or pull in a battle-tested library (etcd/raft, openraft, etc.) and bridge to it?
- Is the log persisted using RocksDB column families inside the existing engine, or a separate store?
- Cross-shard transactions (a workflow needing to write to data that lives in a different shard) — saga at the workflow layer is the v0.1 answer, but call it out.
- Snapshot + log-compaction strategy for long-running workflows.
## Scope of this issue
v0.1 implementation of the consensus lane with per-workflow-shard Raft groups, integrated with the workflow checkpoint store from the step-memoization issue.
---
🤖 Filed by [Claude](https://claude.com/claude-code) on behalf of @kriszyp
Contributor guide
Assessment
This issue has not been assessed yet.