block / block/buzz

idle_timeout kills long in-flight turns despite active turn_liveness

Open
#6,826 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

During a long in-flight ACP turn, `buzz-acp` can emit `turn_liveness` observer frames every ~10s (so Desktop activity shows the turn as alive) while simultaneously counting down `BUZZ_ACP_IDLE_TIMEOUT` based **only** on silence on the agent subprocess stdout. When stdout goes silent longer than `idle_timeout`, the harness kills the turn, respawns the agent subprocess (new ACP session — prior session context orphaned), and surfaces a red activity error: `idle_timeout: Agent session timed out due to inactivity`.

This produces contradictory signals: Desktop sees liveness pings; the harness treats the turn as dead.

## Expected behavior

While a turn is in-flight and `turn_liveness` is actively emitting (or equivalently: while `read_until_response_with_idle_timeout` is waiting on a prompt response), `idle_timeout` should not fire solely because stdout is silent — OR `turn_liveness` should not imply the turn is healthy if stdout silence will kill it.

At minimum, document clearly that:
- `BUZZ_ACP_IDLE_TIMEOUT` resets **only** on valid JSON lines on agent stdout (`acp_read`)
- `turn_liveness` / kind 24200 frames do **not** reset idle
- `buzz messages send` / channel publish does **not** reset idle

## Actual behavior (production incident, 2026-08-25)

**Harness:** `buzz-acp` → `omp acp` (Globis agent, VPS bots)
**Config:** `BUZZ_ACP_IDLE_TIMEOUT=120` (inherited from an earlier OpenClaw anti-churn tuning; default in code is 900s)
**Symptom:** ~30 min visible work in Desktop activity; OMP session map showed active turn at 341k context on `openrouter/stealth/ox-alpha`, then sudden respawn — old session ctx 0, new session with fresh context. Agent had not yet published to channel.

**Journal (bots, `buzz-globis`):**
```
Aug 25 20:24:48 bots buzz-globis[2036161]: WARN buzz_acp::acp: idle timeout (120s) — no agent activity
Aug 25 20:24:48 bots buzz-globis[2036161]: WARN buzz_acp: agent_returned — respawning agent=0 outcome="idle_timeout"
Aug 25 20:24:53 bots buzz-globis[2036161]: INFO buzz_acp: respawn complete agent=0
```

**Mitigation applied:** raised `BUZZ_ACP_IDLE_TIMEOUT` to 3600 (< default `MAX_TURN_DURATION` 7200). Structural fix still needed.

## Code pointers

| Location | Behavior |
|----------|----------|
| `crates/buzz-acp/src/config.rs` | `BUZZ_ACP_IDLE_TIMEOUT` default 900s; resets on agent stdout activity |
| `crates/buzz-acp/src/acp.rs` | `read_until_response_with_idle_timeout`: idle deadline reset on each valid JSON stdout line (`acp_read`); fires `IdleTimeout` on silence |
| `crates/buzz-acp/src/pool.rs` | `run_turn_liveness`: emits `turn_liveness` every `BUZZ_ACP_TURN_LIVENESS_SECS` (default 10) — **does not** extend idle deadline |
| `crates/buzz-acp/src/lib.rs` | On `PromptOutcome::Timeout(Idle)`: respawn agent, emit *"Agent session timed out due to inactivity"*; batch is requeued but ACP session state is lost |

## Proposed fix

1. While a prompt turn is in-flight (waiting in `read_until_response_with_idle_timeout`) **and** liveness task is running, suspend `idle_timeout` — or extend idle deadline on each `turn_liveness` tick (since the harness itself knows the turn is active).

2. Alternatively: stop emitting `turn_liveness` if stdout silence would soon trigger idle kill (honest signal to Desktop).

3. Document `IDLE_TIMEOUT` semantics in `crates/buzz-acp/README.md` (currently mentions 620s default in README vs 900s in code — worth aligning).

## Environment

- buzz-acp from block/buzz main (~Aug 2025)
- Agent adapter: OMP (`omp acp`) — long tool runs can go silent on ACP stdout for extended periods
- Relay: self-hosted, `BUZZ_ACP_RELAY_OBSERVER=true`

## Workaround

Set `BUZZ_ACP_IDLE_TIMEOUT` high enough for longest expected stdout silence (must remain `< BUZZ_ACP_MAX_TURN_DURATION`). Not a substitute for aligned liveness/idle semantics.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.