block / block/buzz

[Bug] Shared-compute agents can run green but never answer on small-context models because relay-mesh max_tokens override is clobbered

Open
#2,558 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

### Summary

On a self-hosted two-node shared-compute (relay-mesh) setup, a managed agent starts, shows green/RUNNING, and receives mentions, but **every agent turn fails** and no reply is ever produced. The cause is a request-sizing fit check: the agent's real request body plus the derived `max_tokens=4096` exceeds the serving model's context window, so the mesh router filters out the only serving host and returns `503 … all 0 tunnel(s)`. The expected mitigation — lowering `BUZZ_AGENT_MAX_OUTPUT_TOKENS` per agent — is accepted and persisted by the UI but has no effect, because `apply_relay_mesh_env` re-sets the value *after* the user/record env is applied.

Direct mesh connectivity was verified working before this finding (small prompts complete through the same ingress), so this is specifically about agent-turn sizing, not transport.

### Environment

- Buzz commit `fd55ab6`, image `ghcr.io/block/buzz:sha-fd55ab6`
- Serving model: `jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M`, context **8192**
- Two nodes: a serving desktop (Share Compute on) and a consumer desktop running the managed agent, connected over the relay-mesh. Lab reproduction.
- OS: Linux on both nodes (headless desktop sessions for the lab reproduction)

### What works (so the failure is isolated to sizing)

- The consumer node joins the mesh; its `/v1/models` lists the SmolLM2 model.
- **Direct inference through the same consumer-side ingress succeeds with small prompts** — e.g. a `max_tokens=5` chat completion returns real model output routed to the serving peer.
- The `buzz-agent` harness and `buzz-acp` bridge run; the agent subscribes to the channel and shows RUNNING.

### Steps to reproduce

1. Serve a small-context model (SmolLM2, 8192 ctx) via Share Compute on one node.
2. On a second node, create a Buzz-shared-compute agent for that model with a normal harness (default tools/MCP) and mention it in a channel.
3. Observe the agent start green and then fail every turn.

### Observed failures (exact strings)

Agent create / mention-triggered start (readiness gate):

```
Buzz shared compute did not become inference-ready for jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M:
HTTP 429 Too Many Requests: {"error":{"message":"model \"jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M\" not currently available — retry later","type":"rate_limit_error","param":null,"code":"rate_limit_exceeded"}}
```

Agent turn:

```
503 Service Unavailable: {"error":{"message":"all 0 tunnel(s) to hosts for Some(\"jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M\") failed (mesh request)","type":"server_error","param":null,"code":"service_unavailable"}}
```

### Root cause (measured)

The agent's actual turn request body — for even a one-line reply, because it includes the system prompt and tool schemas — was measured on the loopback ingress at **21,040 bytes**. The mesh router estimates required context and filters out any host whose context is smaller:

```
ceil(21040 / 4) = 5260 (prompt) + 4096 (completion / max_tokens) + 256 (margin) = 9612 > 8192 (model context)
```

With the only serving host filtered out, there are zero targets and the turn returns `503 … all 0 tunnel(s)`. In this case the serving host exists and is reachable; it is counted as zero because it is filtered out by the context fit check before routing. A control request on the same ingress with a small prompt and `max_tokens=4096` succeeds (200); the same cap with a ~5000-token prompt fails (503) — confirming the fit check as the cause.

Relevant code (Mesh-LLM host runtime, `crates/mesh-llm-host-runtime/src/network/openai/transport.rs`):
- `request_budget_tokens_from_parts` (L1006) — `prompt = ceil(body_len_bytes / 4)`, plus completion tokens, plus margin.
- `request_token_margin` (L1023) / `REQUEST_TOKEN_MARGIN = 256` (L39).
- `reorder_candidates_by_context_and_throughput` (L1075) — drops candidates whose context `< required_tokens`.

### The per-agent override is clobbered

The natural fix is to lower the agent's output cap so `5260 + 1024 + 256 = 6540 < 8192`. Setting `BUZZ_AGENT_MAX_OUTPUT_TOKENS=1024` in the agent's env vars via the Edit-agent UI **is persisted** (`managed-agents.json`: `env_vars: {"BUZZ_AGENT_MAX_OUTPUT_TOKENS":"1024"}`), but the spawned harness still runs with `4096` (verified via `/proc//environ` for both `buzz-acp` and `buzz-agent`, before and after restart).

The spawn path applies env in this order (`desktop/src-tauri/src/managed_agents/runtime.rs`):
- **L1859** — user/record env is applied (`merged_user_env(persona_over_global, record.env_vars)` → `command.env(k, v)`); the surrounding comment states these values "always win".
- **L1869** — `apply_relay_mesh_env` then runs and re-inserts `BUZZ_AGENT_MAX_OUTPUT_TOKENS=4096` via `command.env(k, v)`, **after** the user env, so last-write-wins overwrites the override.

The hardcoded value is in `desktop/src-tauri/src/managed_agents/relay_mesh.rs` (L42, `env.insert("BUZZ_AGENT_MAX_OUTPUT_TOKENS", "4096")`). Net result: for relay-mesh agents there is **no supported config override** for this cap.

### Expected vs actual

- **Expected:** a shared-compute agent on a small-context model either fits its requests to the model's context automatically, or can be made to fit via the per-agent `BUZZ_AGENT_MAX_OUTPUT_TOKENS` env var configured through the UI; and a persistent inability to run surfaces clearly.
- **Actual:** the agent shows green/RUNNING but fails every turn with a generic `503 … all 0 tunnel(s)` (surfaced only in a per-agent log; after retry exhaustion the agent posts a "couldn't process… please re-send" message to the channel), and the per-agent `BUZZ_AGENT_MAX_OUTPUT_TOKENS` override is silently ignored.

### Suggested fixes (the team may prefer a different direction)

1. **Preserve user/record env precedence** — have `apply_relay_mesh_env` only *default* `BUZZ_AGENT_MAX_OUTPUT_TOKENS` (skip if already set by user/record env) so the per-agent env var set through the UI takes effect.
2. **Clamp shared-compute `max_tokens` to the discovered model context** at request-build time (e.g. `min(configured, model_context − estimated_prompt − margin)`), so agents fit small-context models without manual tuning.
3. **Surface this failure clearly** in the agent UI/log/state — distinguish "no serving host fits this request's context" from a generic `503`, rather than showing a green agent that cannot answer.

### Notes

This is a lab reproduction on a private two-node testbed; the team may have a preferred fix direction. Happy to validate a patch on the existing multi-node testbed and report back.

_Redaction: no keys, nsecs, private relay secrets, host IPs, or screenshots with secrets are included._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.