block / block/buzz

Cannot diagnose a missing agent response: no correlation between ACP socket, relay connection, and Redis fan-out

Open
#5,472 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

When an agent does not respond, the logs do not contain enough information to tell where the event stopped. The observable symptom is a WebSocket stream ending, then a reconnect some time later — which looks identical whether the socket was closed cleanly, timed out, was reset, or the event was never fanned out to it in the first place.

## What's missing

Chasing one of these today, the following questions have no answer in the logs:

- **Did the ACP socket close cleanly, time out, reset, or vanish without a Close frame?** The Close code and reason are not preserved, so every ending looks the same.
- **Which relay-side connection handled that exact socket?** There is no identifier shared between the client's view of the socket and the server's, so the two sides' logs cannot be joined.
- **Did Redis report any subscribers when the event was published?** Publishing to zero subscribers and publishing to a subscriber that then dropped the event are indistinguishable.
- **Did the receiving relay match any local WebSocket subscriptions?** Fan-out recipient and drop counts are not recorded.
- **Did ACP receive the event live, or only later via reconnect/replay?** Without receipt age there is no way to tell a slow path from a replayed one.

Each of these is individually small; together they mean the difference between "the agent is broken" and "the event never reached the socket" is not derivable from the logs, and every occurrence turns into a live-reproduction exercise.

## Proposed shape

A correlation trail across the three hops (relay → Redis pub/sub → ACP harness), using IDs and counts only:

- return an opaque connection id on the WebSocket upgrade so ACP-side and relay-side logs can be joined to the same socket
- record ACP connection generations, Close code/reason, and the distinct exit paths (error, EOF, ping timeout) rather than collapsing them
- record event receipt age and a replay hint, so live delivery is distinguishable from replay
- record Redis publish subscriber counts, and relay fan-out recipient/drop counts, keyed on the existing event id
- add Redis subscriber connect/disconnect/reconnect and topic gauges

Two constraints I'd hold this to: it should carry **no message contents, auth material, or keys** — ids, timestamps, and counts only — and it should not change delivery behavior, only describe it.

The one judgment call worth a maintainer's opinion is log volume: making this useful by default means enabling `buzz_pubsub=info` in the relay's default filters, which is a deliberate increase in baseline output. Diagnostics that are off by default tend not to be on when the incident happens, but that trade is the maintainers' call, not mine.

## Status

Filed after the fact: #4769 implements the above and is open. Filing here per CONTRIBUTING.md's issue-first guidance so the approach — particularly the default-log-level question and whether the connection id should be exposed on the upgrade response — can be settled separately from reviewing the diff.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.