HarperFast / HarperFast/harper

Consensus-replicated lane for workflow state

Open
#753 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of #752 (Durable Execution v0.1 epic).

## Problem

Harper's current replication is asynchronous, peer-to-peer, last-writer-wins on timestamp with node-name tiebreak — verified in [\`Table.ts\`](https://github.com/HarperFast/harper/blob/main/resources/Table.ts) at \`precedesExistingVersion\` (around line 4089) and supported by the node-name → node-id mapping in \`nodeIdMapping.ts\`. There is no Raft, no quorum, no leader election in steady state: \`replicateTo\` controls residency and an optional per-request commit-confirmation count, not a quorum write.

This is fine for application data, but unacceptable for workflow execution. A failure mode: node A commits a \`step-7-completed\` checkpoint locally, then crashes before the change propagates. Node B picks up the workflow, doesn't see the checkpoint, re-executes step 7 — and re-charges Stripe. LWW conflict resolution on recovery does not save us; the checkpoint can be lost outright.

## Proposed shape

A **per-shard Raft group on a dedicated workflow-state subsystem, alongside the existing replication path** — not replacing it. Existing data continues to use the current high-throughput peer-to-peer lane; workflow checkpoints and step outputs use a quorum-replicated commit log.

- **Sharding key**: workflow ID. Each workflow has a strict serial order; cross-workflow operations remain eventually consistent.
- **Quorum**: standard Raft majority. A workflow step is not acknowledged committed until a quorum has it.
- **Co-location**: the shard for workflow W lives on the same nodes (or a subset) that hold W's checkpoint data, so the workflow runtime can wake on whichever node holds the leader of W's shard without an extra hop.
- **Leader election + shard placement**: out-of-band; placement should be observable and rebalanceable as load metrics develop.

## Why this is not satisfied by #483

#483 (exclusive record locks) is concurrency control — it prevents two nodes writing the same record at once by broadcasting a lock acquisition. It does **not** make a locally-committed write durable across node failure. The two are complementary: locking ensures a single executor; consensus ensures the executor's checkpoints are durable before the step is acknowledged.

## Open design questions

- Do we use an in-house Raft implementation or pull in a battle-tested library (etcd/raft, openraft, etc.) and bridge to it?
- Is the log persisted using RocksDB column families inside the existing engine, or a separate store?
- Cross-shard transactions (a workflow needing to write to data that lives in a different shard) — saga at the workflow layer is the v0.1 answer, but call it out.
- Snapshot + log-compaction strategy for long-running workflows.

## Scope of this issue

v0.1 implementation of the consensus lane with per-workflow-shard Raft groups, integrated with the workflow checkpoint store from the step-memoization issue.

---

🤖 Filed by [Claude](https://claude.com/claude-code) on behalf of @kriszyp

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.