KeeperHub / KeeperHub/keeperhub

Alternative workflow runtime (Rust or Bun) conforming to the Workflow DevKit protocol

Open
#2,292 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted needs-discussion
Dominant language
TypeScript
Stars
24
Forks
93
Avg merge
1d 4h
Merged PRs (30d)
253

Description

Part of #2294.

## Reason

The workflow engine runs on the Vercel Workflow DevKit: `workflow` 4.8.5 and `@workflow/world-postgres` 4.3.5 are direct dependencies, durable state lives in the Postgres world (`pnpm db:setup-workflow` runs `@workflow/world-postgres/bin/setup.js`), and the per-execution runner is `keeperhub-executor/workflow-runner.ts`, started under `tsx` inside a Job container.

We are open to a different runtime underneath that - a Rust implementation, or Bun - provided it speaks the same protocol and the same Postgres world, so this is a runtime swap rather than a fork.

**Read the other sub-issues on the tracking issue before starting.** The two costs we can currently point at (a mean 5s fixed jitter, and a cold Kubernetes Job per execution) sit outside the runtime, so a faster runtime is upside on top of those rather than a substitute for them. That is a sequencing note, not a rejection: this issue is open and the work is wanted.

## Scope

An alternative runtime for workflow execution that conforms to the Workflow DevKit protocol and the `@workflow/world-postgres` durable-state format.

Two shapes are in scope, and they are very different sizes:

- **Bun.** Smallest delta. The engine and the plugin steps stay TypeScript; the question is measured gain and whether everything survives - the `node:vm`-based sandbox path, `postgres-js`, and the `tsx` bootstrap in particular.
- **Rust.** Largest delta, and the interesting one. See the crux below.

## The crux for a Rust implementation

Every plugin step in this repository is TypeScript (`plugins/`), and there are a lot of them. A Rust runtime therefore has to do one of:

1. Drive the existing TypeScript steps, embedding a JS engine and paying its cost at the boundary
2. Reimplement steps in Rust, which is a rewrite of the plugin surface rather than a runtime change
3. Handle only the durable-execution and orchestration layer in Rust, leaving step bodies where they are

State which of the three you are proposing in your first comment. They have different costs, different risks, and only the third is plausibly incremental.

## What a contribution needs to show

- **Protocol conformance, demonstrated not asserted.** The same workflow definition must produce the same durable state transitions in the Postgres world, and existing execution rows written by the current runtime must remain readable. A runtime that cannot resume an execution the current one started is a fork.
- **Measured before and after.** A profiling framework already exists at `scripts/runtime/workflow_runtime_analysis/` (static analysis, V8 step profiling, whole-workflow profiling, WASM fuel calibration). Use it rather than building a new one, and note its own recorded finding that static complexity does not predict runtime cost.
- **Which surface is replaced**, stated explicitly: step execution only, orchestration only, or the whole engine.
- **The sandbox story.** User JavaScript from the Code node runs in a separate hardened service (`sandbox/`). Say whether your runtime changes that boundary. It must not weaken it.
- **A migration path.** Both runtimes will run concurrently during any rollout, against one Postgres world.

## Note on the ordering

If #2289 has landed, use its numbers to state what fraction of end-to-end latency your change actually addresses. If it has not, measure the runner stage locally and show the figure. A runtime change justified by interpreter benchmarks rather than by workflow execution timings will be hard to review.

---
Tracking: [KEEP-1271](https://linear.app/keeperhubapp/issue/KEEP-1271) (internal tracker, not publicly accessible)

Contributor guide

Open the contributing guide

Research direction

Read the related sub-issues first, then inspect keeperhub-executor/workflow-runner.ts, plugins/, sandbox/, and scripts/runtime/workflow_runtime_analysis/. Choose and state one Rust approach or evaluate Bun, measure the runner and workflow timings, and compare protocol state transitions and resume behavior against the existing Postgres world. Done means demonstrated conformance, preserved sandbox isolation, explicit replacement scope, and a concurrent-runtime migration path.

Written by the indexing model from the issue text.

Assessment

Tech stack
bun, kubernetes, node.js, postgresql, rust, typescript
Domain
backend, databases, devops, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.