Relay-mesh agents silently stop replying if mesh runtime drops after launch (no self-heal)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
Relay-mesh (Buzz shared compute) agents **silently stop replying** if the mesh runtime drops at any point after app launch. No error surfaces to the user and nothing re-arms the runtime until an agent is manually (re)started.
## Repro
1. Launch desktop with a relay-mesh agent; mesh runtime comes up, ingress on `127.0.0.1:9337`, bot replies fine.
2. Leave the app running for an extended period (overnight), or otherwise have the embedded mesh runtime exit/wedge.
3. Send the bot a message → **no reply, no error**.
4. Manually start/create an agent → `ensure_relay_mesh_for_record` runs again, re-arms `9337`, replies resume.
Observed live: `9337/3131 NOT listening` while `goose acp` agent subprocesses were still running and healthy; `curl 127.0.0.1:9337/v1/models` → connection refused. A remote peer (M4) was serving the model the whole time, so compute availability was never the issue.
## Root cause
`ensure_relay_mesh_for_record` (which brings up the local mesh client node / OpenAI ingress) is only invoked:
- at launch — `managed_agents/restore.rs` (`restore_managed_agents_on_launch`)
- on agent create/start — `commands/agents.rs` (`start_local_agent_with_preflight`)
There is **no periodic re-arm and no liveness check** tying the mesh runtime to agent message dispatch. If `state.mesh_llm_runtime` becomes `None` (runtime exited, or a bad restart left a zombie — cf. `"timed out after 5s waiting for embedded mesh runtime thread to exit"`), relay-mesh agents keep running but point at a dead ingress. The failure is invisible.
## Impact
Any long-lived desktop session using shared-compute agents will eventually see bots go quiet with no indication why. This is the dominant "shared compute is broken" user-facing symptom even though discovery, transport, and remote serving are all healthy.
## Suggested fix (options)
1. **On-demand re-arm:** before dispatching a message to a relay-mesh agent (or in the agent reconcile loop), if `mesh_llm_runtime.is_none()` or the ingress is unreachable, call `ensure_relay_mesh_for_record` / `ensure_client_node_for_model` to bring it back.
2. **Liveness + surfaced error:** health-check the runtime periodically; if it's down and can't be restored (e.g. no live serving member), surface a visible "shared compute offline" state on the agent instead of silent non-reply.
3. **Zombie guard:** the `mesh runtime thread to exit` 5s timeout can leave a half-dead runtime — ensure a failed restart doesn't leave `mesh_llm_runtime` in a state that blocks re-arm.
Fix (1) alone resolves the reported symptom; (2) makes it debuggable; (3) prevents the wedge that motivated it.
## Not this
- Not a transport/discovery bug (those were fixed in #2024, present in main).
- Not remote-serving availability (peer was serving throughout).
Contributor guide
Research direction
Trace ensure_relay_mesh_for_record from managed_agents/restore.rs and commands/agents.rs, then follow how relay-mesh agent message dispatch relates to mesh_llm_runtime. Reproduce the runtime-drop scenario and verify that a long-lived agent resumes replying after recovery, or that an unrecoverable runtime failure is surfaced instead of remaining silent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- desktop, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100