HarperFast / HarperFast/harper
[Epic] Durable Execution v0.1
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Tracking issue for a Harper-native durable execution engine: a workflow runtime where step results are checkpointed, workflows survive crashes via replay, signals/sleeps survive node restarts, and — Harper's structural differentiator — workflow checkpoints commit *atomically* with multi-model application data inside a single Harper transaction.
Source proposal: design doc (Durable Execution on Harper) — happy to attach on request, but the salient details are below and broken out into child issues.
## Why
Every major agent framework (LangGraph 1.0, LlamaIndex Workflows, Pydantic AI, OpenAI Agents SDK + Temporal) has converged on durable execution as a baseline for production AI agent workloads. Harper's pitch is unique here: because the database engine and application/Resource layer run in the same Node.js process, the workflow checkpoint can ride inside the same LMDB/RocksDB transaction as the user's table writes. Temporal cannot do this. DBOS can do it only when every write is SQL against the same Postgres-compatible database. For multi-model agent data, Harper would be the only credible option.
## Architectural position
**Two compounding benefits:**
1. **Atomic commits between workflow state and multi-model application data** when data lives in Harper.
2. **Lower engine overhead per step** regardless of where data lives — every step transition is in-process function calls + a local commit, not several network round trips.
These support both \"system of record stays in Harper\" and \"system of record stays elsewhere; agent working data + workflow state in Harper\" deployments.
## Scope of v0.1
New-core primitives that *cannot* be approximated on top of existing Harper primitives:
- [ ] HarperFast/harper#753 — Consensus-replicated lane for workflow state (per-shard Raft)
- [ ] HarperFast/harper#754 — Native durable timer service (sharded hierarchical timing wheel)
- [ ] HarperFast/harper#755 — Step memoization with idempotency keys
- [ ] HarperFast/harper#756 — Determinism contract for workflow code (SES Compartment policy)
- [ ] HarperFast/harper#757 — Workflow developer API surface (Workflow base class, ctx.step, ctx.atomic, ctx.signal, ctx.sleep)
## Reusable as-is (not separate issues)
The proposal identified primitives already present:
- \`transaction(fn)\` in [resources/transaction.ts](https://github.com/HarperFast/harper/blob/main/resources/transaction.ts) for multi-statement atomic writes within a database — direct foundation for \`ctx.atomic\`.
- \`Context\` propagation via \`AsyncLocalStorage\` — the natural carrier for workflow context.
- SES Compartment support in [security/jsLoader.ts](https://github.com/HarperFast/harper/blob/main/security/jsLoader.ts#L87) — substrate for the determinism sandbox.
## POC-tier (do inline during prototyping)
Per scope decision, no separate tickets for POC-tier items now — they will be assembled on existing primitives during v0.1 prototyping and replaced with optimized core primitives as benchmarks expose ceilings:
- Commit-log subscription via \`RocksTransactionLogStore\` \`'aftercommit'\` hook → needs a workflow-aware fast-path filter at scale
- Point-in-time replay via the existing audit-log + \`getRecordAtTime\` → needs a dedicated workflow event store with workflow-ID-keyed indexes and its own retention
- Signal delivery via the Resource subscription mechanism → may need a workflow-specific signal primitive for one-shot wake-and-resume routing
- Scheduler as a polling Resource on \`(fireAt, workflowId, stepId, payload)\` rows → replaced by the durable timer service (#754)
## v0.2 follow-ups
- **Cross-database atomicity for workflows that span databases.** Multi-database writes inside \`transaction(fn)\` are chained sequentially with no 2PC ([\`DatabaseTransaction.this.next\`](https://github.com/HarperFast/harper/blob/main/resources/DatabaseTransaction.ts)) — partial-write hazard if the workflow checkpoint and user data live in different DBs. v0.1 answer: workflows scoped to a single database. v0.2 needs 2PC or a saga primitive at the workflow layer.
## Alignment with related work
- **#626 (Built-in Harper Agent Component)** — the built-in agent is a natural first internal consumer once the engine exists. Its \`schedule_followup\` tool (currently \`setTimeout(...).unref()\` on main thread) should ride the durable timer service (#754). Its \`hdb_agent_session\` + \`pendingApprovals\` are bespoke versions of \`ctx.step\` checkpointing + \`ctx.signal\`. HarperFast/harper-pro#676 should ship independently; revisit when v0.1 lands.
- **#483 (Exclusive record locks)** — complementary, not a substitute. Locking solves \"single executor\" for a workflow; consensus (#753) solves \"checkpoint is durable before we ack the step.\" The runtime probably wants both.
## What this is not solving
- **Universal cross-system atomicity.** External calls (LLM providers, Stripe, third-party APIs) remain at-least-once with idempotency keys.
- **A turnkey Temporal replacement at scale.** Different sweet spot: workloads where the atomic-commit-with-data guarantee is decisive.
- **Multi-tenant managed workflows.** Workflows for untrusted multi-tenant scenarios require hardening SES Compartment isolation or process-level isolation — out of scope for initial release.
## Open product questions
- Design partner first vs. spec first (proposal recommends design partner before v0.1 lands).
- Declarative graph DSL in addition to imperative TS workflows?
- User control over sharding granularity?
- Pricing/packaging — Harper feature, separate tier, or Fabric-only?
- Determinism strictness — mechanical SES enforcement vs. convention + warnings?
---
🤖 Filed by [Claude](https://claude.com/claude-code) on behalf of @kriszyp
Contributor guide
Assessment
This issue has not been assessed yet.