KeeperHub / KeeperHub/keeperhub
feat: resume a partially failed payout run without re-paying settled legs
- Dominant language
- TypeScript
- Stars
- 24
- Forks
- 93
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 253
Description
### Before filing
- [x] I searched open and closed issues for this proposal.
- [x] I checked the docs and the current behaviour on `staging`.
- [x] This is one change, not several. (Several means several issues.)
Source re-checked against `staging` at `75f25e7`; the behaviour below was observed on app.keeperhub.com.
### Reason: what you cannot do today
A payout run that fails partway through cannot be resumed. Re-running it starts again from the first leg, so every recipient who was already paid gets paid again.
**Observed** on app.keeperhub.com, Base Sepolia, 2026-09-15. One workflow: Manual trigger → For Each (`concurrency: "sequential"`) → `web3/transfer-token` (USDC `0x036CbD53842c5426634e7929541eC2318f3dCF7e`), over `[{recipient: R1, amount: "1"}, {recipient: R2, amount: "1"}]`. The org wallet held exactly 1 USDC, so leg 2 had to fail. Balances and `Transfer` events below were read from chain, not from KeeperHub.
| | R1 | R2 | Org | USDC Transfers from org |
|---|---|---|---|---|
| Before run 1 | 0 | 0 | 1.0 | |
| After run 1 (`8om5c2jiw0jyoxk2k8npq`, `error`) | 1.0 | 0 | 0 | → R1 [0x5195…7725](https://sepolia.basescan.org/tx/0x5195212356269cb9f21cb3dd0c4b56497f264cc63298052974a9a45286157725) |
| Top-up of exactly 1.0 USDC; same workflow re-run, byte-identical input | 1.0 | 0 | 1.0 | |
| After run 2 (`9hqtgyzgmop197zs882zl`, `error`) | **2.0** | **0** | 0 | → R1 [0x75d0…5b04](https://sepolia.basescan.org/tx/0x75d06301acc68bf6b2839c2da4ca741707397fe3a2b75032c730217c12865b04) |
Both times, leg 2 failed before broadcast with `Insufficient USDC balance. Have: 0.0, Need: 1`, and no transaction from the org reached R2. Run 2 sent leg 1 again, which used up the top-up, so it never reached R2. The recipient already paid got paid twice; the one still owed got nothing.
R1 `0x106175F175B940CcA1816d75eB19937a88BE7720`, R2 `0x9cBa0Ef1D7CC6B78e46F060F551D175DA45Aa98a`, org wallet `0xE6e5e247ce27A43F724675DD679DC7a4a1896CA6`, workflow `cfob296j2poa5allyf8bf`.
Raw GET /api/workflows/executions/{id}/status for both runs, unedited
Run 1 (`8om5c2jiw0jyoxk2k8npq`):
```json
{"status":"error","nodeStatuses":[{"nodeId":"trigger-1","status":"success"},{"nodeId":"transfer-1","status":"success"},{"nodeId":"transfer-1","status":"error"},{"nodeId":"foreach-1","status":"error"}],"progress":{"totalSteps":3,"completedSteps":4,"runningSteps":0,"currentNodeId":null,"currentNodeName":null,"percentage":133},"errorContext":{"failedNodeId":null,"lastSuccessfulNodeId":"transfer-1","lastSuccessfulNodeName":"Transfer Token","executionTrace":["trigger-1","foreach-1","transfer-1","transfer-1"],"error":"Insufficient USDC balance. Have: 0.0, Need: 1"},"transactionHashes":[{"hash":"0x5195212356269cb9f21cb3dd0c4b56497f264cc63298052974a9a45286157725","nodeId":"transfer-1","chainId":84532,"nodeName":"Transfer Token","iterationIndex":0}]}
```
Run 2 (`9hqtgyzgmop197zs882zl`):
```json
{"status":"error","nodeStatuses":[{"nodeId":"trigger-1","status":"success"},{"nodeId":"foreach-1","status":"error"},{"nodeId":"transfer-1","status":"success"},{"nodeId":"transfer-1","status":"error"}],"progress":{"totalSteps":3,"completedSteps":4,"runningSteps":0,"currentNodeId":null,"currentNodeName":null,"percentage":133},"errorContext":{"failedNodeId":null,"lastSuccessfulNodeId":"transfer-1","lastSuccessfulNodeName":"Transfer Token","executionTrace":["trigger-1","foreach-1","transfer-1","transfer-1"],"error":"Insufficient USDC balance. Have: 0.0, Need: 1"},"transactionHashes":[{"hash":"0x75d06301acc68bf6b2839c2da4ca741707397fe3a2b75032c730217c12865b04","nodeId":"transfer-1","chainId":84532,"nodeName":"Transfer Token","iterationIndex":0}]}
```
**From source** (`staging` @ `75f25e7`), why:
- An execution walks the DAG from the trigger (`lib/workflow/executor/executor.workflow.ts:2348`). The only record of completed steps is in memory, per execution (`lib/workflow/executor/step-success-tracker.ts:2`, `:72`), and is cleared by `clearExecution` (`:202`). Nothing starts a For Each from a previous execution's iterations.
- A failed iteration never stops the other iterations (`lib/workflow/nodes/for-each/concurrency.ts:28`), and the loop then fails closed (`executor.workflow.ts:2214`). That is correct within one run. It leaves the paid/unpaid split recoverable only by reading logs.
- Transfer steps are never retried (`plugins/web3/steps/transfer-token.ts:37`, `transfer-funds.ts:59`). A re-run is the only way to recover, and the re-run is what double-pays.
- Passing an `idempotency_key` does not help: `execute_workflow` forwards it to `POST /api/workflow/{id}/execute` (`lib/mcp/tools.ts:1202`), which scopes it to `workflow-execute:${workflowId}` and for 24 hours replays the original executionId whether that run completed or failed (`app/api/workflow/[workflowId]/execute/route.ts:350-357`, `lib/idempotency.ts:17`, `:206`), so reusing the key returns the failed run and pays no one, a new key starts a fresh execution from the first leg, and nothing under `lib/workflow/` consumes a key.
- The step claims from #2465 cover neither the run nor its legs. They are keyed on `(execution_id, node_id)` and skip For Each bodies by design (`lib/workflow/executor/step-claim.ts:14-25`, `:83`). A re-run is a new execution_id.
For Each around `transfer-funds` follows the same executor path. It was not run; that claim is from source.
### Reason: what the workaround costs
To resume safely today, an operator has to work out which legs settled, trust that the failed legs never broadcast, and build a new input by hand with only the unpaid legs. Each step has a trap:
- **The status response can't tell you which legs settled.** Observed in both runs: `nodeStatuses` lists `transfer-1` twice (`success`, `error`) with no iteration index, `errorContext.failedNodeId` is `null`, and `progress.percentage` is `133`. The per-iteration detail exists only in `get_execution` logs, one execution at a time, and nothing links run 2 to run 1.
- **Under gas sponsorship, the org wallet's nonce doesn't count legs.** Observed: leg 1 was an EIP-7702 transaction sent by relayer `0x6331eb4571de9284f7e9ead98ac7b0661a091e99`. The org's nonce went 0 → 1 without the org sending anything, and the org address now carries the 7702 delegation. So a "did we already send?" check based on the org nonce gives the wrong answer.
- **A leg whose broadcast outcome is unknown must not be sent again,** and nothing today records that state across executions.
The public template catalogue has a "Monthly Salary Distribution" template (`search_templates`, id `nhggiz2hn76gbqhiw9d2z`) that pays five contractor addresses through five `web3/transfer-funds` nodes. Every workflow that pays a list either rebuilds this bookkeeping itself or accepts the risk of double-paying. The fix that looks obvious, re-running, pays the wrong person first.
### Scope: what this touches, and what it does not
**In**
- A new action, working name `web3/disburse`. It pays an explicit list of legs `{recipient, amount}` in sequence, on one network and one asset per node. The caller supplies a `runKey` that names the payout run across executions.
- A new table holding each leg's state, keyed `(organization_id, run_key, leg_index)`. This needs a migration.
- Per-leg results in the node's output.
- Assets: native (EVM and Solana, both through `transfer-funds-core`; the Solana path is at `plugins/web3/steps/transfer-funds-core.ts:605-618`), ERC-20 through `transfer-token-core`, SPL through `transfer-spl-token-core`. Each leg reuses the existing core, so signing, sponsorship, nonce sessions and receipt verification stay as they are.
- Spend limits: native legs keep reserving against the cap one leg at a time through `withStepValueCap` (`transfer-funds.ts:45`). A resumed run must not reserve again for legs that already settled.
**Solana** is in scope for the new action. The resume logic depends only on the claim rows and receipts, not on the chain, and the transfer cores already support Solana. However:
- The experiment above was EVM-only (Base Sepolia, ERC-20). No Solana runtime behaviour is claimed as observed.
- Simulation is EVM-only per the MCP docs ("simulation is EVM-only; Solana chain IDs 101/103 and their aliases are rejected before the API call", `lib/mcp/tools.ts:1709`; `docs/FAQ.md:37`). `readContract` also rejects on Solana (`lib/web3/chain-adapter/solana.ts:433`). So correctness must never depend on simulating or contract-reading a leg first. Solana balance checks go through the adapter, as `transfer-funds-core` already does.
- If triage prefers EVM first with Solana as a follow-up, the table and the resume rules don't change.
**Out**
- Atomic all-or-nothing payouts. That is `batch-write-contract` (#1970).
- Changing For Each, or extending #2465's claims into loop bodies. That needs the full nesting path first (`step-claim.ts:19-20`).
- Running legs in parallel. v1 is sequential.
- Safe and Role signer routing.
- Metering ERC-20 legs against the daily cap. `transfer-token` isn't metered today either; that's a separate question.
- Making the status route iteration-aware. That's a separate issue.
- Several networks in one node.
**Surfaces checked**
- For Each + `transfer-token`: reproduced above.
- For Each + `transfer-funds`: same executor path; not run.
- Direct `POST /api/execute/transfer`: one transfer per request, with a per-request Idempotency-Key and no multi-leg state.
- `batch-write-contract`: one atomic transaction; EVM-only, EOA-only, at most 200 calls (`batch-write-contract-core.ts:71`, `:607`); withholds per-call results when the outcome is unknown (`:757`).
- Workflow execute `Idempotency-Key`: replays the old run or starts over, as described above.
### Plan: what you propose
**Config:** `network`; the asset (native, a `tokenConfig` as in `transfer-token`, or an SPL mint); `runKey` (a required string, e.g. an off-chain payroll batch id); `legs` (an array or a template reference).
**Table `disbursement_legs`:** primary key `(organization_id, run_key, leg_index)`, plus `recipient, amount, asset, chain_id, status, transaction_hash, execution_id, claimed_at, settled_at`. Status is one of `claimed | settled | failed | unknown`. Each leg is claimed with one `INSERT … ON CONFLICT DO UPDATE … WHERE claimed_at < stale`. That is the same pattern `workflow_step_claims` uses (#2465, `step-claim.ts:128-146`), but keyed on the run rather than the execution, so it survives a re-run. If #2465's helpers turn out to generalise cleanly, I'll build on them rather than duplicate them.
**What each run does per leg:**
- `settled` (receipt status 1 verified): skip it and report `already_paid` with the original hash. No transfer, and no spend-cap reservation.
- `failed` for certain (rejected before broadcast, e.g. insufficient balance, or a confirmed revert): send it again.
- `unknown` (broadcast, outcome not confirmed): don't send. Check the stored hash's receipt first. If the outcome is still unknown, report `unknown` and fail the node closed.
- Same `run_key` and `leg_index` but a different recipient, amount or asset: reject the whole run with a conflict, rather than paying a list that has changed.
**Output:** `results: [{index, recipient, amount, status: paid | already_paid | failed | unknown, transactionHash?, error?}]`, plus counts. `success` is true only when every leg is `paid` or `already_paid`. Every hash from the run goes into `transactionHashes`, so the existing receipt verification covers every leg.
**Where I need triage input:**
1. How should one node that sends several transactions fill `transactionHashes`? `TransactionHashEntry` has `iterationIndex` (`lib/db/schema.ts:661-667`) but no leg index. Should we reuse it, or add a field?
2. Should `run_key` be required? I'd keep it required: a key derived automatically can't tell "resume this run" from "pay this list again next month".
3. How long should `disbursement_legs` rows be kept?
Existing callers are unaffected: this adds a new action and a new table.
### Plan: alternatives you considered
1. **Use `batch-write-contract` (#1970).** Its calls share one Multicall3 `aggregate3` transaction, so a batch is atomic, but only within that single transaction. It is EVM-only, EOA-only and capped at 200 calls. It takes arbitrary calldata rather than recipient/amount legs, and when the transaction's outcome is unknown it deliberately withholds per-call results (`batch-write-contract-core.ts:757`). It doesn't address resuming a partial failure: its failures are all-or-nothing, and re-running a batch whose outcome is unknown carries the same double-pay risk for the whole batch. It also can't serve Solana.
2. **Pass `idempotency_key` to `execute_workflow`.** Reusing the key replays the failed run; a new key starts again from the first leg (see Reason).
3. **Extend #2465's claims into For Each bodies.** Its claims are keyed on execution_id. Even once the nesting path exists, a re-run is a new execution. #2465 deduplicates replays inside one execution, not payout legs across executions.
4. **Keep For Each and do the resume bookkeeping inside the workflow,** e.g. a Database Query node that filters out paid recipients. That's possible, but every author rebuilds it, it has to read receipts correctly (including sponsored sends), and it can't be atomic with the send.
5. **Do nothing.** Cost: every list-paying workflow that fails partway leaves the operator to rebuild the paid set by hand, and the obvious fix, re-running, double-pays. As observed above, it pays the wrong recipient first.
### Scope: compatibility
- [ ] Changes an existing response shape, status code, CLI flag, or default.
- [ ] Adds, removes, or upgrades a dependency.
- [x] Changes database schema or requires a migration.
- [x] Touches authentication, permissions, validation, or spend limits.
- [ ] Changes pricing, plan limits, or anything a user is charged.
Contributor guide
Research direction
Start with the existing transfer cores in plugins/web3/steps/transfer-funds-core.ts, transfer-token-core.ts, and transfer-spl-token-core.ts, then read lib/workflow/executor/step-claim.ts and lib/db/schema.ts. Trace how transaction hashes and spend-cap reservations are recorded before defining the new disburse action and migration. Done means resumed runs preserve per-leg state, skip settled legs safely, report each result, and cover the stated EVM and Solana paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, blockchain, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100