Why did four simple agent replies take 13–31 seconds?
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Why this issue exists
Four simple agent turns on 2026-07-22 took **13–31 seconds** from the published
mention to the published reply. Every turn completed, but the outer event
timestamps cannot tell us where the time went.
This is a small directional sample, not a performance benchmark. It is enough
to justify stage-level timing so we can separate relay delivery, queueing,
runtime startup, model work, and reply publication.
## What I measured
The turns were sequential and requested only a short acknowledgement, with no
external lookup. Times are relay event `created_at` values with one-second
resolution.
| Turn | Mention time (UTC) | Reply time (UTC) | Elapsed |
| --- | ---: | ---: | ---: |
| 1 | 16:11:23 | 16:11:43 | 20s |
| 2 | 16:12:24 | 16:12:55 | 31s |
| 3 | 16:13:03 | 16:13:28 | 25s |
| 4 | 16:14:57 | 16:15:10 | 13s |
For these four turns:
- mean: 22.25s
- median: 22.5s
- range: 13–31s
## What this tells us
All four mentions were delivered and answered. Equivalent trivial turns still
varied by 18 seconds.
It does **not** identify the slow stage. The elapsed time can include:
1. relay fan-out and harness receipt;
2. queue or admission delay;
3. harness or ACP runtime wake-up;
4. provider/model time to first output and completion;
5. reply publication from the harness;
6. relay acceptance and delivery of the reply.
This also does not replace the reliability bugs where an agent never receives
or processes a turn.
## What #2408 covers
#2408 adds process-local timing from harness relay receipt through reply
fan-out, including queue admission, ACP session resolution, prompt dispatch,
first semantic output, and turn completion.
That is a useful first slice, but it does not yet measure sender publication,
relay delivery before harness receipt, relay acceptance after fan-out, or
recipient rendering. It also does not establish hosted warm/cold baselines,
latency budgets, or a scheduled regression job, so this issue should remain
open after #2408 lands.
## What the full measurement needs
Use one correlation identifier and monotonic timestamps at these boundaries:
1. triggering event accepted by the relay;
2. event received by `buzz-acp`;
3. event admitted to the turn queue;
4. ACP child ready, with cold and warm paths separated;
5. first ACP/model output;
6. turn complete;
7. reply publication initiated;
8. reply accepted by the relay.
Report both stage durations and total latency. Performance telemetry must not
record message content, prompts, credentials, model output, or tool arguments.
Benchmark at least:
- warm harness with a warm ACP/model session;
- warm harness with a cold or lazy ACP start;
- app or harness restart recovery;
- local runtime versus relay-mesh runtime;
- no-tool acknowledgement versus a tool-using turn.
## Done when
- [ ] A repeatable benchmark covers the full mention-to-reply path.
- [ ] Results record sample size, p50, p95, and max rather than only averages.
- [ ] Warm and cold paths are reported separately.
- [ ] The team records latency budgets for each supported path.
- [ ] CI or a scheduled performance job detects material regressions.
- [ ] Telemetry contains correlation and timing metadata only.
## Related work
- #2270 / #2375: mention-gated agents miss untagged replies in joined threads.
- #2062 / #2304: relay-mesh agents stop replying after local ingress failure.
- #1743 / #1862 / #2296: offline mentions can disappear without reliable
sender feedback.
- #1467: removes one reply-sending relay round trip when the root is cached.
- #1263: desktop GUI latency harness; it does not measure agent-turn latency.
- #2122: warm harness sockets with lazy LLM startup.
Those issues should stay focused on missed turns, dead runtimes, and delivery
feedback. This issue asks a different question: when the agent does reply, where
did the time go?
Contributor guide
Research direction
Start by reading the process-local timing described in #2408 and the existing harness relay path. Define a repeatable full mention-to-reply benchmark covering the listed boundaries and warm/cold scenarios, with p50, p95, and max results. Done means timing-only telemetry, latency budgets, and CI or a scheduled regression job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, observability, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100