HarperFast / HarperFast/harper

Step memoization with idempotency keys for workflow replay

Open
#755 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of #752 (Durable Execution v0.1 epic).

## Problem

The replay model for durable workflows requires that on crash recovery (or any replay), completed steps return their cached output instead of re-executing. Without this, the workflow re-invokes every step — including external calls — every time it resumes.

## Proposed shape

A **workflow-aware checkpoint store** with a schema specialized for the workflow runtime's access pattern:

| Column | Purpose |
|---|---|
| \`workflowId\` | Partition key, matches consensus shard from #753 |
| \`stepId\` | Stable per-step identifier (the name passed to \`ctx.step\` / \`ctx.atomic\`) |
| \`idempotencyKey\` | Stable per-logical-invocation key the runtime passes into the step body via \`memo.idempotencyKey\` |
| \`inputHash\` | Hash of step inputs for detecting accidental non-determinism on replay |
| \`output\` | Serialized step result, returned on replay |
| \`completedAt\` | Completion timestamp |
| \`status\` | \`pending\` / \`completed\` / \`failed\` |
| \`error\` | Serialized failure (for replay of failure semantics) |

## Integration with existing primitives

- Commits ride inside the existing \`transaction(fn)\` from [\`resources/transaction.ts\`](https://github.com/HarperFast/harper/blob/main/resources/transaction.ts) — this is what gives \`ctx.atomic\` its differentiator: the workflow checkpoint and the user's table writes commit as one LMDB/RocksDB transaction.
- For \`ctx.step\` (non-atomic): the checkpoint is its own small transaction, separate from anything the step body did in external systems.
- For \`ctx.atomic\`: the checkpoint write is *spliced into* the user-supplied \`transaction(fn)\` so both commit together.

## Retention

The global audit log retention default is 24 hours ([\`auditStore.ts:109\`](https://github.com/HarperFast/harper/blob/main/resources/auditStore.ts#L109)). Workflow checkpoints need their own retention policy — independent, configurable, defaulting to something workflow-lifecycle-appropriate (workflow completes → checkpoints retained N days, then GC'd; running workflows → retained indefinitely).

## Indexes

- Primary: \`(workflowId, stepId)\` for replay lookup.
- Secondary: \`(workflowId, completedAt)\` for transcript reconstruction.
- Possibly tertiary: \`(idempotencyKey)\` for cross-workflow dedup if we expose it as an API.

## Idempotency-key generation

The runtime needs a stable key per logical step invocation that is identical across replays. Proposed derivation: \`hash(workflowId || stepId || invocationCount)\` for in-loop steps, with the engine providing the value to step bodies as \`memo.idempotencyKey\` for forwarding to external services (the doc's email/LLM examples).

## Open questions

- Hard size cap per checkpoint output? Large LLM responses or vector blobs are common — paginate, or store large outputs in a separate blob ref?
- Compression of output payloads on commit?
- Cross-workflow idempotency-key namespace (engine-internal vs. user-supplied for at-most-once external calls)?

## Out of scope

Step memoization for steps that span databases — that requires cross-DB atomicity which is the v0.2 follow-up in #752.

---

🤖 Filed by [Claude](https://claude.com/claude-code) on behalf of @kriszyp

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.