KeeperHub / KeeperHub/keeperhub
Ultrafast workflow runtime: reduce end-to-end trigger-to-broadcast latency
- Dominant language
- TypeScript
- Stars
- 24
- Forks
- 93
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 266
Description
Tracking issue. The work is broken into the four issues checklisted below; discussion of the overall direction belongs here, implementation discussion belongs on the individual issues.
## Reason
Pre-inclusion triggers (#2239) are only worth building if the reaction path is short enough for the lead time to survive. It currently is not, and the largest costs we can point at today sit outside the workflow runtime rather than inside it.
Stage costs, read from code:
| Stage | Cost | Source |
| -- | -- | -- |
| `eth_getLogs` drain against confirmed blocks | rate limited to one request per second per chain | `keeperhub-events/event-tracker/src/chains/provider-manager.ts:108` |
| Random sleep before forwarding a matched event to SQS | uniform 0 to 10s, mean 5s | `keeperhub-events/event-tracker/src/listener/event-listener.ts:29,200-202` |
| SQS enqueue plus executor receive | not measured | |
| Fresh Kubernetes Job per execution | pod schedule, image pull check, Node boot, bootstrap | `EXECUTION_MODE: "isolated"`, `deploy/keeperhub-stack/prod/values.yaml:1065`; `keeperhub-executor/execution-mode.ts` |
| Workflow steps, remote signing, submit | not measured | |
Two of those five rows say "not measured", and the others are read from source rather than observed. That is the first problem, and it is why the first sub-issue is measurement rather than optimisation.
The two rows that are legible are both large. The deliberate jitter alone has a mean of 5s, which is longer than the entire pre-inclusion window on a 2s-block chain. And in production every execution takes a fresh Kubernetes Job, including workflows that do nothing but read a contract and post to Discord.
## A note on ordering, because runtimes are the usual first suggestion
Replacing Node with Bun, or moving the engine to Rust, does not address either legible cost above. A fixed multi-second sleep and a cold pod per execution are unaffected by how fast the interpreter is.
That is a statement about ordering, not a rejection. An alternative runtime is genuinely wanted and is tracked as its own sub-issue below - the constraint is that it must conform to the Workflow DevKit protocol and the `@workflow/world-postgres` durable-state format we already run on, so that it is a runtime swap rather than a fork. Read that issue before starting, and expect to justify the change with workflow execution timings rather than interpreter benchmarks.
## Scope
Reduce measured time from trigger observed to transaction broadcast. Four pieces:
1. Measure the path end to end
2. Replace the fixed jitter with load spreading that does not add a fixed mean delay
3. Stop paying for isolation on executions that do not need it
4. An alternative runtime (Rust or Bun) conforming to the existing workflow protocol
**Not in scope:** widening what can trigger a workflow, which is #2239. The two are complementary - that issue is worthless without this one for tier 4, and this one is useful on its own.
## Sub-issues
- [ ] #2289 - End-to-end trigger-to-broadcast latency is inferred from code, never measured
- [ ] #2290 - Every event-triggered execution waits a mean of 5s on a fixed random jitter before SQS
- [ ] #2291 - Every execution pays for a cold isolated Kubernetes Job, including those that need no isolation
- [ ] #2292 - Alternative workflow runtime (Rust or Bun) conforming to the Workflow DevKit protocol
---
Tracking: [KEEP-1271](https://linear.app/keeperhubapp/issue/KEEP-1271) (internal tracker, not publicly accessible)
Contributor guide
Research direction
Start with the four linked sub-issues, especially #2289, and read the mentioned provider-manager.ts, event-listener.ts, deploy/keeperhub-stack/prod/values.yaml, and keeperhub-executor/execution-mode.ts entries. The overall work is done when trigger-to-broadcast latency is measured and the four latency-reduction areas are addressed while preserving the Workflow DevKit protocol and @workflow/world-postgres format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, bun, kubernetes, node.js, rust, typescript
- Domain
- backend, devops, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100