KeeperHub / KeeperHub/keeperhub

End-to-end trigger-to-broadcast latency is inferred from code, never measured

Open
#2,289 1 comment 0 reactions 0 assignees View on GitHub
accepted enhancement good first issue help wanted
Dominant language
TypeScript
Stars
24
Forks
93
Avg merge
1d 4h
Merged PRs (30d)
253

Description

Part of #2294.

## Reason

Nobody can currently say how long KeeperHub takes to react to an on-chain event, because the path is not instrumented end to end. The stage table on the parent issue is read from source, and two of its five stages have no figure at all.

Every decision downstream of this depends on the answer. Whether the jitter matters more than the cold pod, whether pre-inclusion triggers are worth building at all, whether a runtime change would help - all of those are currently arguments from code structure rather than from data, and they can be settled by measurement.

## Scope

Carry a timestamp and a correlation id through the pipeline, and record the interval at each hand-off:

- Event observed by the event-tracker (and the block timestamp it came from, so chain-side lag is separable from ours)
- Forwarded to SQS
- Received by the executor
- Execution started, whichever dispatch target was chosen
- Transaction broadcast

Report as a histogram per stage, split by dispatch target (`k8s-job` versus `in-process`), so a slow stage is attributable rather than averaged away.

## What a solution needs to cover

- The event-tracker, the executor and the runner are separate services. The correlation id has to survive the SQS boundary and the Job boundary, and it should reuse whatever tracing identity already exists rather than adding a parallel one.
- Cardinality. Per-chain and per-dispatch-target labels are useful; per-workflow labels on a latency histogram are a metrics-cost problem. Say which labels you chose and why.
- Chain-side lag is not our latency, but it is needed to interpret ours. Keep them separable.

## What "done" looks like

A dashboard, or a query someone can run, that answers: for a given chain over the last 24 hours, what was the distribution of time from event observed to transaction broadcast, and which stage dominated it.

Post the first real numbers on this issue when you have them. They decide the priority of the other two sub-issues, and they may well contradict the ordering assumed on the parent.

---
Tracking: [KEEP-1271](https://linear.app/keeperhubapp/issue/KEEP-1271) (internal tracker, not publicly accessible)

Contributor guide

Open the contributing guide

Research direction

Start by locating the event-tracker, executor, and runner entry points, then trace the existing tracing identity across the SQS and Job boundaries. Define stage timestamps and labels without adding workflow-level histogram cardinality, and identify the dashboard or query path. Done means a 24-hour per-chain distribution from event observation to broadcast with stage breakdowns and initial real numbers.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, typescript
Domain
backend, distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.