Cannot diagnose a missing agent response: no correlation between ACP socket, relay connection, and Redis fan-out
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
When an agent does not respond, the logs do not contain enough information to tell where the event stopped. The observable symptom is a WebSocket stream ending, then a reconnect some time later — which looks identical whether the socket was closed cleanly, timed out, was reset, or the event was never fanned out to it in the first place.
## What's missing
Chasing one of these today, the following questions have no answer in the logs:
- **Did the ACP socket close cleanly, time out, reset, or vanish without a Close frame?** The Close code and reason are not preserved, so every ending looks the same.
- **Which relay-side connection handled that exact socket?** There is no identifier shared between the client's view of the socket and the server's, so the two sides' logs cannot be joined.
- **Did Redis report any subscribers when the event was published?** Publishing to zero subscribers and publishing to a subscriber that then dropped the event are indistinguishable.
- **Did the receiving relay match any local WebSocket subscriptions?** Fan-out recipient and drop counts are not recorded.
- **Did ACP receive the event live, or only later via reconnect/replay?** Without receipt age there is no way to tell a slow path from a replayed one.
Each of these is individually small; together they mean the difference between "the agent is broken" and "the event never reached the socket" is not derivable from the logs, and every occurrence turns into a live-reproduction exercise.
## Proposed shape
A correlation trail across the three hops (relay → Redis pub/sub → ACP harness), using IDs and counts only:
- return an opaque connection id on the WebSocket upgrade so ACP-side and relay-side logs can be joined to the same socket
- record ACP connection generations, Close code/reason, and the distinct exit paths (error, EOF, ping timeout) rather than collapsing them
- record event receipt age and a replay hint, so live delivery is distinguishable from replay
- record Redis publish subscriber counts, and relay fan-out recipient/drop counts, keyed on the existing event id
- add Redis subscriber connect/disconnect/reconnect and topic gauges
Two constraints I'd hold this to: it should carry **no message contents, auth material, or keys** — ids, timestamps, and counts only — and it should not change delivery behavior, only describe it.
The one judgment call worth a maintainer's opinion is log volume: making this useful by default means enabling `buzz_pubsub=info` in the relay's default filters, which is a deliberate increase in baseline output. Diagnostics that are off by default tend not to be on when the incident happens, but that trade is the maintainers' call, not mine.
## Status
Filed after the fact: #4769 implements the above and is open. Filing here per CONTRIBUTING.md's issue-first guidance so the approach — particularly the default-log-level question and whether the connection id should be exposed on the upgrade response — can be settled separately from reviewing the diff.
Contributor guide
Assessment
This issue has not been assessed yet.