block / block/buzz

buzz-acp: idle-pool reaper tears down the pool right after a long turn, orphaning background tasks

Open
#7,171 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

The idle-pool reaper clocks in `buzz-acp` (idle-pool sleep and exit-after-inactivity) are refreshed only at **dispatch** time. A turn that runs longer than the idle bound is torn down on the first reaper tick after it completes: the sidecar kills the CLI process together with its live background tasks and any wake-ups the agent scheduled for after the turn.

## Observed incident

Production setup with `--idle-pool-sleep 900`. An agent ran a 28-minute turn (long build/verification). The turn completed successfully at 17:59:08, and the pool was reaped 13 seconds later on the next reaper tick — `last_activity` still pointed at the dispatch 28 minutes earlier. Two background tasks the agent was waiting on were orphaned; their completion never woke the agent, which looked like the agent silently going dark for 45+ minutes.

## Mechanism

- `last_activity` is refreshed when a prompt is dispatched into the pool, but not when a `PromptResult` comes back or when a steer is acked.
- Any turn longer than the idle bound therefore *guarantees* the pool looks idle the moment it finishes.
- Teardown does not account for live background tasks of the CLI process, and there is no resume/re-wake path after the pool is gone.

## Suggested fix

Count turn completion and steer acks as activity when refreshing the idle clocks. Heartbeat results should stay excluded — heartbeat dispatch is gated on an idle slot with no flushable work, so counting them would keep a heartbeat-enabled pool awake forever.

I have a patch running in production (18 agents) since 2026-09-01 with no premature teardowns after long turns; PR incoming.

## Known adjacent edges (out of scope for the minimal fix)

- The in-flight deadline is `max_turn_duration + 100s`; a turn exceeding it auto-expires and the pool can still be reaped mid-turn.
- The panic path does not refresh the clocks.
- Background tasks and scheduled wake-ups still don't keep an otherwise-idle pool alive on their own; that needs a separate awareness/resume mechanism.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.