block / block/buzz

Relay-mesh agents silently stop replying if mesh runtime drops after launch (no self-heal)

Open
#2,062 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

Relay-mesh (Buzz shared compute) agents **silently stop replying** if the mesh runtime drops at any point after app launch. No error surfaces to the user and nothing re-arms the runtime until an agent is manually (re)started.

## Repro

1. Launch desktop with a relay-mesh agent; mesh runtime comes up, ingress on `127.0.0.1:9337`, bot replies fine.
2. Leave the app running for an extended period (overnight), or otherwise have the embedded mesh runtime exit/wedge.
3. Send the bot a message → **no reply, no error**.
4. Manually start/create an agent → `ensure_relay_mesh_for_record` runs again, re-arms `9337`, replies resume.

Observed live: `9337/3131 NOT listening` while `goose acp` agent subprocesses were still running and healthy; `curl 127.0.0.1:9337/v1/models` → connection refused. A remote peer (M4) was serving the model the whole time, so compute availability was never the issue.

## Root cause

`ensure_relay_mesh_for_record` (which brings up the local mesh client node / OpenAI ingress) is only invoked:

- at launch — `managed_agents/restore.rs` (`restore_managed_agents_on_launch`)
- on agent create/start — `commands/agents.rs` (`start_local_agent_with_preflight`)

There is **no periodic re-arm and no liveness check** tying the mesh runtime to agent message dispatch. If `state.mesh_llm_runtime` becomes `None` (runtime exited, or a bad restart left a zombie — cf. `"timed out after 5s waiting for embedded mesh runtime thread to exit"`), relay-mesh agents keep running but point at a dead ingress. The failure is invisible.

## Impact

Any long-lived desktop session using shared-compute agents will eventually see bots go quiet with no indication why. This is the dominant "shared compute is broken" user-facing symptom even though discovery, transport, and remote serving are all healthy.

## Suggested fix (options)

1. **On-demand re-arm:** before dispatching a message to a relay-mesh agent (or in the agent reconcile loop), if `mesh_llm_runtime.is_none()` or the ingress is unreachable, call `ensure_relay_mesh_for_record` / `ensure_client_node_for_model` to bring it back.
2. **Liveness + surfaced error:** health-check the runtime periodically; if it's down and can't be restored (e.g. no live serving member), surface a visible "shared compute offline" state on the agent instead of silent non-reply.
3. **Zombie guard:** the `mesh runtime thread to exit` 5s timeout can leave a half-dead runtime — ensure a failed restart doesn't leave `mesh_llm_runtime` in a state that blocks re-arm.

Fix (1) alone resolves the reported symptom; (2) makes it debuggable; (3) prevents the wedge that motivated it.

## Not this

- Not a transport/discovery bug (those were fixed in #2024, present in main).
- Not remote-serving availability (peer was serving throughout).

Contributor guide

Open the contributing guide

Research direction

Trace ensure_relay_mesh_for_record from managed_agents/restore.rs and commands/agents.rs, then follow how relay-mesh agent message dispatch relates to mesh_llm_runtime. Reproduce the runtime-drop scenario and verify that a long-lived agent resumes replying after recovery, or that an unrecoverable runtime failure is surfaced instead of remaining silent.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
desktop, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.