KeeperHub / KeeperHub/keeperhub

Every execution pays for a cold isolated Kubernetes Job, including those that need no isolation

Open
#2,291 0 comments 0 reactions 0 assignees View on GitHub
enhancement needs-discussion
Dominant language
TypeScript
Stars
24
Forks
93
Avg merge
1d 4h
Merged PRs (30d)
253

Description

Part of #2294. This one touches a security boundary. Read all of it before starting.

## Reason

Production runs `EXECUTION_MODE: "isolated"` (`deploy/keeperhub-stack/prod/values.yaml:1065`), which means every execution gets a fresh Kubernetes Job: pod scheduling, image pull check, Node boot, bootstrap - before the first workflow step runs. A workflow that reads a contract and posts to Discord pays exactly the same cost as one that signs and broadcasts a transaction.

This is both the largest legible latency term and a direct compute cost, so it is the one change on this tracking issue that improves both.

The alternative already exists in code. `resolveDispatchTarget` (`keeperhub-executor/execution-mode.ts`) supports a `complex` mode that routes only workflows containing a web3 write to a Job and runs everything else in-process. It is not what production uses.

## Scope

Determine whether `complex` is safe to run in production, and if it is not as written, define what would be.

## Why this is not simply a config change

The mode selector decides an isolation boundary. Getting the classifier wrong does not produce a bug, it produces a workflow running with less isolation than it needed, in a multi-tenant system. Specifically:

- The current signal is `requiresCredentials` on plugin action definitions. That flag is overloaded and does not cleanly mean "this execution handles secret material". Whether it is the right signal is the central question of this issue, not an assumption to build on.
- Web3 writes are not the only reason to isolate. A Code node runs user-supplied JavaScript. It already has its own out-of-process sandbox (`sandbox/`), so it may not need the Job as well - but that argument has to be made explicitly, not skipped.
- Any node handling a decrypted integration credential is in the same category regardless of whether it writes to a chain.
- The classifier runs against the workflow definition. A workflow edited between dispatch and execution, or one whose node types are resolved dynamically, must not be able to downgrade its own placement.

## What a solution needs to cover

- A written statement of what the isolated Job protects against, and which workflow properties imply that protection is needed. The classifier follows from that; it should not be reverse-engineered from the current implementation.
- Tests that pin the classification for each category, including a workflow that mixes categories, which must take the stricter path.
- A staged rollout path. This is not a flag flip in production on merge.
- Measured latency and cost difference for the workflows that change placement.

## Note

A well-argued written answer of "the current classifier is not sound, and here is the one that is" is a complete and welcome contribution to this issue, with or without the implementation.

---
Tracking: [KEEP-1271](https://linear.app/keeperhubapp/issue/KEEP-1271) (internal tracker, not publicly accessible)

Contributor guide

Open the contributing guide

Research direction

Start by reading keeperhub-executor/execution-mode.ts and deploy/keeperhub-stack/prod/values.yaml:1065, then inspect the plugin action definitions and sandbox/ boundary. Establish what the isolated Job protects and whether complex is safe, adding classification tests for each category and mixed workflows. Done includes a staged rollout plan and measured latency and cost changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, typescript
Domain
backend, infrastructure, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.