HarperFast / HarperFast/harper
Step memoization with idempotency keys for workflow replay
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
Part of #752 (Durable Execution v0.1 epic).
## Problem
The replay model for durable workflows requires that on crash recovery (or any replay), completed steps return their cached output instead of re-executing. Without this, the workflow re-invokes every step — including external calls — every time it resumes.
## Proposed shape
A **workflow-aware checkpoint store** with a schema specialized for the workflow runtime's access pattern:
| Column | Purpose |
|---|---|
| \`workflowId\` | Partition key, matches consensus shard from #753 |
| \`stepId\` | Stable per-step identifier (the name passed to \`ctx.step\` / \`ctx.atomic\`) |
| \`idempotencyKey\` | Stable per-logical-invocation key the runtime passes into the step body via \`memo.idempotencyKey\` |
| \`inputHash\` | Hash of step inputs for detecting accidental non-determinism on replay |
| \`output\` | Serialized step result, returned on replay |
| \`completedAt\` | Completion timestamp |
| \`status\` | \`pending\` / \`completed\` / \`failed\` |
| \`error\` | Serialized failure (for replay of failure semantics) |
## Integration with existing primitives
- Commits ride inside the existing \`transaction(fn)\` from [\`resources/transaction.ts\`](https://github.com/HarperFast/harper/blob/main/resources/transaction.ts) — this is what gives \`ctx.atomic\` its differentiator: the workflow checkpoint and the user's table writes commit as one LMDB/RocksDB transaction.
- For \`ctx.step\` (non-atomic): the checkpoint is its own small transaction, separate from anything the step body did in external systems.
- For \`ctx.atomic\`: the checkpoint write is *spliced into* the user-supplied \`transaction(fn)\` so both commit together.
## Retention
The global audit log retention default is 24 hours ([\`auditStore.ts:109\`](https://github.com/HarperFast/harper/blob/main/resources/auditStore.ts#L109)). Workflow checkpoints need their own retention policy — independent, configurable, defaulting to something workflow-lifecycle-appropriate (workflow completes → checkpoints retained N days, then GC'd; running workflows → retained indefinitely).
## Indexes
- Primary: \`(workflowId, stepId)\` for replay lookup.
- Secondary: \`(workflowId, completedAt)\` for transcript reconstruction.
- Possibly tertiary: \`(idempotencyKey)\` for cross-workflow dedup if we expose it as an API.
## Idempotency-key generation
The runtime needs a stable key per logical step invocation that is identical across replays. Proposed derivation: \`hash(workflowId || stepId || invocationCount)\` for in-loop steps, with the engine providing the value to step bodies as \`memo.idempotencyKey\` for forwarding to external services (the doc's email/LLM examples).
## Open questions
- Hard size cap per checkpoint output? Large LLM responses or vector blobs are common — paginate, or store large outputs in a separate blob ref?
- Compression of output payloads on commit?
- Cross-workflow idempotency-key namespace (engine-internal vs. user-supplied for at-most-once external calls)?
## Out of scope
Step memoization for steps that span databases — that requires cross-DB atomicity which is the v0.2 follow-up in #752.
---
🤖 Filed by [Claude](https://claude.com/claude-code) on behalf of @kriszyp
Contributor guide
Assessment
This issue has not been assessed yet.