block / block/buzz

Why did four simple agent replies take 13–31 seconds?

Open
#2,386 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Why this issue exists

Four simple agent turns on 2026-07-22 took **13–31 seconds** from the published
mention to the published reply. Every turn completed, but the outer event
timestamps cannot tell us where the time went.

This is a small directional sample, not a performance benchmark. It is enough
to justify stage-level timing so we can separate relay delivery, queueing,
runtime startup, model work, and reply publication.

## What I measured

The turns were sequential and requested only a short acknowledgement, with no
external lookup. Times are relay event `created_at` values with one-second
resolution.

| Turn | Mention time (UTC) | Reply time (UTC) | Elapsed |
| --- | ---: | ---: | ---: |
| 1 | 16:11:23 | 16:11:43 | 20s |
| 2 | 16:12:24 | 16:12:55 | 31s |
| 3 | 16:13:03 | 16:13:28 | 25s |
| 4 | 16:14:57 | 16:15:10 | 13s |

For these four turns:

- mean: 22.25s
- median: 22.5s
- range: 13–31s

## What this tells us

All four mentions were delivered and answered. Equivalent trivial turns still
varied by 18 seconds.

It does **not** identify the slow stage. The elapsed time can include:

1. relay fan-out and harness receipt;
2. queue or admission delay;
3. harness or ACP runtime wake-up;
4. provider/model time to first output and completion;
5. reply publication from the harness;
6. relay acceptance and delivery of the reply.

This also does not replace the reliability bugs where an agent never receives
or processes a turn.

## What #2408 covers

#2408 adds process-local timing from harness relay receipt through reply
fan-out, including queue admission, ACP session resolution, prompt dispatch,
first semantic output, and turn completion.

That is a useful first slice, but it does not yet measure sender publication,
relay delivery before harness receipt, relay acceptance after fan-out, or
recipient rendering. It also does not establish hosted warm/cold baselines,
latency budgets, or a scheduled regression job, so this issue should remain
open after #2408 lands.

## What the full measurement needs

Use one correlation identifier and monotonic timestamps at these boundaries:

1. triggering event accepted by the relay;
2. event received by `buzz-acp`;
3. event admitted to the turn queue;
4. ACP child ready, with cold and warm paths separated;
5. first ACP/model output;
6. turn complete;
7. reply publication initiated;
8. reply accepted by the relay.

Report both stage durations and total latency. Performance telemetry must not
record message content, prompts, credentials, model output, or tool arguments.

Benchmark at least:

- warm harness with a warm ACP/model session;
- warm harness with a cold or lazy ACP start;
- app or harness restart recovery;
- local runtime versus relay-mesh runtime;
- no-tool acknowledgement versus a tool-using turn.

## Done when

- [ ] A repeatable benchmark covers the full mention-to-reply path.
- [ ] Results record sample size, p50, p95, and max rather than only averages.
- [ ] Warm and cold paths are reported separately.
- [ ] The team records latency budgets for each supported path.
- [ ] CI or a scheduled performance job detects material regressions.
- [ ] Telemetry contains correlation and timing metadata only.

## Related work

- #2270 / #2375: mention-gated agents miss untagged replies in joined threads.
- #2062 / #2304: relay-mesh agents stop replying after local ingress failure.
- #1743 / #1862 / #2296: offline mentions can disappear without reliable
sender feedback.
- #1467: removes one reply-sending relay round trip when the root is cached.
- #1263: desktop GUI latency harness; it does not measure agent-turn latency.
- #2122: warm harness sockets with lazy LLM startup.

Those issues should stay focused on missed turns, dead runtimes, and delivery
feedback. This issue asks a different question: when the agent does reply, where
did the time go?

Contributor guide

Open the contributing guide

Research direction

Start by reading the process-local timing described in #2408 and the existing harness relay path. Define a repeatable full mention-to-reply benchmark covering the listed boundaries and warm/cold scenarios, with p50, p95, and max results. Done means timing-only telemetry, latency budgets, and CI or a scheduled regression job.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.