KeeperHub / KeeperHub/keeperhub

feat(triggers): fire workflows on reverted and internal calls by wiring the trace matcher into the event tracker

Closed
#2,464 4 comments 0 reactions 0 assignees View on GitHub
accepted confirmed enhancement
Dominant language
TypeScript
Stars
24
Forks
93
Avg merge
1d 8h
Merged PRs (30d)
266

Description

### Before filing

- [x] I searched open and closed issues for this proposal.
- [x] I checked the docs and the current behaviour on `staging`.
- [x] This is one change, not several. (Several means several issues.)

### Reason: what you cannot do today

A workflow cannot trigger on anything a contract did not emit as an event. #2241 set out the cases: a reverted drain attempt against a monitored contract, an internal ETH transfer, a `delegatecall` into an unlogged implementation, and an unlogged privileged call on a third-party contract. None of them produce a log, so none of them reach the Event trigger. #2239 lists this as Tier 2 of trigger observability.

The pieces for it are on `staging`, but nothing connects them:

- #2393 merged `matchTraceCalls` in `lib/web3/trace-decode.ts`. It filters `callTracer` frames on caller, callee, selector, call type, minimum value and revert status. Nothing imports it, and #2241 closed when it merged. Its PR listed the remaining work as trigger-type wiring, event-tracker integration and provider access.
- #2273 and #2277 surveyed which upstreams serve `debug_traceBlockByNumber` (`.planning/issue-2247-trace-upstream-survey.md`).
- `a27bd3b` (#2240) wired the state-threshold trigger into `ChainProviderManager`'s drain loop. A trace trigger fits the same shape.

Checked on `staging` at `af80267`.

### Reason: what the workaround costs

There is no workaround inside KeeperHub. The alternatives are running a separate trace indexer or a third-party monitor and forwarding its alerts to a Webhook trigger, which moves the security-relevant part of the workflow off-platform. Failing that, you poll `debug_traceTransaction` from a Schedule trigger, which cannot find a transaction you don't already have the hash for.

### Scope: what this touches, and what it does not

Touches:

1. **Event tracker (`keeperhub-events/event-tracker`)**: a `Trace` registration, listener, and trace fetch on the existing drain loop. Dark, the way #2240's tracker slice shipped: nothing reaches it until the app admits the trigger type.
2. **App**: a `Trace` value in `WorkflowTriggerEnum`, admission in `/api/workflows/events`, a config panel in `trigger-config.tsx`, the MCP `TRIGGERS` schema and trigger-input schema, template outputs, and a docs page.

Does not touch:

- The Event, Transfer, Block, or state-threshold trigger paths, apart from one fix in the shared block-staleness watchdog, noted in the plan.
- Solana. `debug_trace*` does not exist there.
- The database schema. No migration.
- Upstream configuration and plans. No new RPC dependencies or keys.
- Pre-inclusion (mempool) observation (#2243–#2246) and state thresholds (#2240).

On "one change": this is one issue with two PRs, in the order #2240 used. The tracker slice ships alone and is correct but inert. The app slice depends on it. Neither is a separate feature.

### Plan: what you propose

Tracker slice. I have this built and tested on a branch, and will open it once this is accepted.

- **Where traces are fetched.** Inside `ChainProviderManager.drain`, one `debug_traceBlockByNumber` request (callTracer) per block, shared by every trace subscription on the chain. It covers the same contiguous range the log path serves, not a head sample: a revert in a block the drain did not trace would never be seen.
- **The mark.** Traces share the high-water mark, so a range is served once both its logs and its traces are. Traces therefore inherit the reorg rewind and the catch-up bound. On a chain with trace subscribers, one drain spans at most 10 blocks, because traces cost a request per block.
- **Upstream support is learned, not configured.** Per #2247 it varies by upstream and plan, not by chain. A refusal (`-32601`, or the refusal bodies the survey recorded) marks the connection unsupported, logs once, and reports the range served. An untraceable upstream therefore cannot pin the mark and stall log delivery. Support is re-learned on reconnect, because a reconnect may land on the fallback URL. Transient errors leave the block owed.
- **Dedup identity.** `trace:{workflowId}:{chainId}:{txHash}:{frameIndex}`, held by the phantom-execution unique index. It needs no arming state: every match is a discrete frame of a transaction. Accepted gap: if a reorg re-includes a transaction and it executes differently, a frame can move to a new index and dispatch twice.
- **Burst bound.** Dispatches are capped per subscription per block (25), and the overflow is logged with its count.
- **Transaction hashes.** Entries without `txHash` (Geth before 1.13) are paired with `eth_getBlockByNumber`'s transaction list by position. An untraceable transaction inside a block is skipped and counted, without costing the rest of the block.
- **Node config** (the tracker's `RawWorkflowNodeConfig`):
- `triggerType: "Trace"`, `network`, and `contractAddress` as the watched callee (required, so a trace trigger is never chain-wide);
- optional `traceCaller`;
- `traceSelector`, or `abiFunction` + `contractABI` resolved to a selector;
- `traceCallTypes`, `traceMinValueWei`, and `traceStatus` (`success` default | `reverted` | `any`).
- Every field is validated at map time. An unparseable filter field refuses the workflow rather than turning into a wildcard.
- **Trigger data.** `triggerType`, `chainId`, `blockNumber`, `transactionHash`, `transactionIndex`, `frameIndex`, `callType`, `from`, `to`, `value` (decimal wei), `selector`, `input`, `depth`, `reverted`.
- **The watchdog fix.** The block-staleness watchdog now checks for any subscriber kind. It checked log subscribers only, so a trace-only or state-only chain whose socket stopped delivering blocks was never caught.
- **Matcher code.** The flattening and selector rules are copied from `lib/web3/trace-decode.ts`, because the package cannot import the monorepo root. This follows the precedent `multicall3.ts` documents.

Evidence so far:

- 57 new unit tests, and the existing 273 still pass. They cover shared tracing per block, contiguous ranges, a failed block staying owed, an unsupported upstream not pinning the mark and logs still flowing, the hash fallback, the span cap, mapper refusals, dispatch keys, duplicate and refused dispatches, and the per-block cap.
- Plasma mainnet over HTTP: 12 blocks (76 txs, 1,512 frames) parsed with none skipped. The matcher selected the reverted frames.
- Tempo mainnet: the real `ChainProviderManager` over `wss://rpc.tempo.xyz` traced 63 consecutive blocks in 40 s with no gaps. This settles WSS as a workable transport on at least one chain the survey marked viable.

App slice, after the tracker slice:

- `WorkflowTriggerEnum.TRACE = "Trace"`, admitted in `app/api/workflows/events/route.ts`.
- A `trigger-config.tsx` panel:
- network, and watched contract with ABI auto-fetch so a function can be picked instead of a raw selector;
- optional caller;
- a call-type multi-select;
- minimum value entered in the native unit and stored as wei;
- status.
- `TRIGGERS` in `app/api/mcp/schemas/route.ts`, `lib/mcp/trigger-input-schema.ts`, and template outputs so `{{Trigger.transactionHash}}`, `{{Trigger.from}}` and similar autocomplete.
- A public docs page stating plainly which networks serve traces today.

Questions for triage:

1. @zkasuran, you wrote the matcher and listed the wiring as next. Are you already on it? If so, I will stand down or split along whatever line suits you.
2. **Production upstreams.** The survey could not see `CHAIN_RPC_CONFIG`. Until someone confirms production serves `debug_*`, the capability check keeps the tracker safe on every chain. For the UI, should the network list show every EVM chain with a "requires a tracing upstream" note, or only chains known to trace?
3. **Where that chain list lives.** A static list in the app, or a column on `chains`? The column needs a migration, so I would rather not add one without a yes.
4. **Transport.** The tracker slice fetches over the chain's existing WSS connection, and Tempo mainnet works. If production WSS upstreams cap frame size below a dense block's trace, the fallback would be the chain's HTTP RPC for this one call. I propose leaving that until a real cap is observed.

### Plan: alternatives you considered

- **A separate trace service or satellite process.** Rejected, for the reason #2240 settled on the drain loop: a second connection, cadence and mark per chain, for no gain.
- **Sampling traces at the head, like state triggers.** Rejected: it misses the events this trigger exists for.
- **A separate trace mark.** Rejected for now. It would need its own reorg rewind and catch-up handling. The shared mark gets both, at the cost of re-fetching a range's logs when its traces fail, which the dispatch key already absorbs.
- **`trace_block` (Parity-style) as an alternative method.** Deferred. Plasma and Tempo both serve `debug_traceBlockByNumber`, and a second parser only pays off for an upstream that serves `trace_block` alone.
- **Importing `lib/web3/trace-decode.ts` directly.** Not possible: the package's `rootDir` cannot resolve modules above itself.
- **Doing nothing.** Leaves #2393's matcher unused.

### Scope: compatibility

- [ ] Changes an existing response shape, status code, CLI flag, or default.
- [ ] Adds, removes, or upgrades a dependency.
- [ ] Changes database schema or requires a migration.
- [ ] Touches authentication, permissions, validation, or spend limits.
- [ ] Changes pricing, plan limits, or anything a user is charged.

Contributor guide

Open the contributing guide

Research direction

Start with lib/web3/trace-decode.ts and the ChainProviderManager.drain entry point, then inspect the keeperhub-events/event-tracker tests covering the proposed tracker slice. Review app/api/workflows/events/route.ts, trigger-config.tsx, the MCP schema files, and template outputs for the app slice. Done means trace subscriptions, admission, configuration, trigger data, tests, and the public docs page work together without changing the database schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend, blockchain, documentation, frontend, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.