KeeperHub / KeeperHub/keeperhub

Nothing a workflow computes survives the run: no workflow-scoped key-value state

Open
#2,288 2 comments 0 reactions 0 assignees View on GitHub
accepted enhancement help wanted
Dominant language
TypeScript
Stars
24
Forks
93
Avg merge
1d 8h
Merged PRs (30d)
266

Description

Part of #2293. Read the design questions there before starting.

## Reason

Nothing a workflow computes survives the run. There is no workflow-scoped key-value store anywhere in the schema, so a monitoring workflow that needs a cursor - "the last block I scanned", "the transactions I have already alerted on" - has two options:

1. Re-scan a rolling time window on every run and re-derive what is new. This has a visible failure mode: a resolved alert can reopen when the same underlying transaction is still inside the window on the next pass.
2. Provision an external database and reach it through the Database Query node. That works, and it is disproportionate for one key-value pair.

Most workflow platforms ship a small built-in store for exactly this reason. KeeperHub does not have one.

## Scope

Two actions, workflow-scoped:

- `state/get` - `{ key }` returns `{ value, exists }`
- `state/set` - `{ key, value, ttl? }` returns `{ success }`

Backed by storage KeeperHub already operates, so the user provisions nothing.

## What a solution needs to cover

- **Scope and isolation.** Keyed by workflow, isolated per organization. A workflow must not be able to read another workflow's state, and the key namespace must not be shared across organizations.
- **Concurrency.** Two executions of the same workflow can overlap. Read-modify-write on the same key will race. Either the operations are atomic, or the race is documented and the API offers something that is (compare-and-set, or an atomic increment).
- **Size and count limits, enforced.** This is a per-workflow store, not a database. An unbounded store attached to every workflow is a storage-growth problem with no natural ceiling.
- **TTL and eviction**, so abandoned keys do not accumulate forever.
- **Deletion semantics.** What happens to a workflow's state when the workflow is deleted, duplicated, or exported and imported elsewhere.
- **Visibility.** Whether the current state is inspectable from the UI. Debugging a monitor with an opaque cursor is unpleasant; this does not have to ship in the first version, but the storage choice should not rule it out.

## Note on the storage choice

The title of the internal ticket behind this says "serverless-first, DB as fallback", and that framing is worth keeping: the point is that the user provisions nothing. Whether the backing store is Postgres or Redis is an implementation decision, and it should be argued from the concurrency and eviction requirements above rather than assumed.

## Out of scope

Cross-workflow shared state, and state readable from outside an execution through the API. Both are reasonable later; neither should be designed into a first version.

---
Tracking: [KEEP-1036](https://linear.app/keeperhubapp/issue/KEEP-1036) (internal tracker, not publicly accessible)

Contributor guide

Open the contributing guide

Research direction

Start by reading the design questions in #2293, which this issue identifies as prerequisite context. No source files, tests, or implementation entry points are named. Done means defining and implementing workflow-scoped get/set actions with isolation, concurrency behavior, limits, TTL and lifecycle semantics, while keeping cross-workflow and external API access out of scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.