[Bug]: Memory exhaustion compounds through three paths — unbounded stream buffers, keep-alive instance stacking, and doom-loop gate evasion (controlled repro + minimal fixes)

## QwenPaw Version

v2.2.0 (official `agentscope/qwenpaw:latest` image, `io.qwenpaw.managed-runtime-boundary=2.2.0`)

## Description

Container memory exhaustion (fills at ~1MB/s, then the service hangs/OOMs) is not one bug but **three compounding paths**. #7222-style slow growth is only path C; the two faster paths below are unreported, and **strong models mask them** — which may be why the fast paths are hard to reproduce and #4265/#2992 have stalled.

| Path | Mechanism | Trigger | Scale |
|---|---|---|---|
| A. Loop | context mismatch + varied-args tool loop evading DoomLoopGate | weak model + default config + big files | seconds–minutes |
| B. Stream | infra slowdown stretches run lifetime; replay buffers/queues unbounded; failures silent | flaky/slow relay | minutes–hours |
| C. Keep-alive | reload with active runs stacks old workspace instances (24h cap, stuck tasks never cancelled) | model switch while runs active | hours–days |

**Real-production reproduction (most stable recipe)**: weak self-hosted infra — Qwen3.6-27B served by vLLM on 4× L20 GPUs; QwenPaw v2.2.0 launched with container limits (**4 cores / 8GB memory**); default context settings (**max_input 131072 / max_token 8192** while the vLLM endpoint actually serves 8192 — a 16× window mismatch); agent runs long-horizon heavy tasks (image-review workflows are the most reliable trigger). This configuration reproduces **~1MB/s memory fill followed by a hung service** fairly consistently.

**Related PR(s):** #7723 (first of a series; working fixes for all six items exist and are measured — see the comment below)

**Security considerations:** none (memory/streaming behavior; no auth or config exposure)

## Component(s) Affected

- [x] Core / Backend (app, agents, config, providers, utils, local_models)
- [ ] Console (frontend web UI)
- [ ] Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
- [ ] Skills
- [ ] CLI
- [ ] Documentation (website)
- [ ] Tests
- [ ] CI/CD
- [ ] Scripts / Deploy

## Environment

- **QwenPaw version:** v2.2.0 official image
- **OS:** Ubuntu 22.04 host; Docker containers (2C/4G for the controlled rig, 4C/8G for the production repro)
- **Install method:** Docker (`agentscope/qwenpaw:latest`)
- **Python version:** 3.11 (image venv)
- **Models:** controlled rig — mock OpenAI-compatible server + real `deepseek-v4-flash` via a fault-injecting proxy; production repro — Qwen3.6-27B on vLLM (4× L20)

---

## Details (code anchors, controlled reproduction, and suggested fixes)

## Summary

#7222 describes slow unbounded RSS growth (20GB over 2 days). During a controlled reproduction
campaign (official `v2.2.0` image, 2C/4G container, mock model + fault-injecting proxy + real
DeepSeek-flash), we confirmed **three distinct paths** that compound into the reported
"container memory fills at >1MB/s" signature. #7222 covers only the slow path; the two faster
paths below are unreported, and **strong models mask them** — which may be why the fast paths
have been hard to reproduce and #4265/#2992 stalled.

| Path | Mechanism | Trigger | Scale |
|---|---|---|---|
| A. Loop | context mismatch + varied-args tool loop evading DoomLoopGate | weak model + default config + big files | seconds–minutes |
| B. Stream | infra slowdown stretches run lifetime; replay buffers/queues unbounded; failures silent | flaky/slow relay | minutes–hours |
| C. Keep-alive | reload with active runs stacks old workspace instances (24h cap, stuck tasks never cancelled) | model switch while runs active | hours–days |

**Real-production reproduction (most stable recipe)**: weak self-hosted infra — Qwen3.6-27B served by vLLM on 4× L20 GPUs; QwenPaw v2.2.0 launched with container limits (**4 cores / 8GB memory**); default context settings (**max_input 131072 / max_token 8192** while the vLLM endpoint actually serves 8192 — a 16× window mismatch); agent runs long-horizon heavy tasks (image-review workflows are the most reliable trigger). This configuration reproduces **~1MB/s memory fill followed by a hung service** fairly consistently.

## Path A — loop gate evasion (fast)

- `get_model_max_input_length` falls back to **128K** for models missing from the catalog
(`config.py`), so compaction never fires before the real (smaller) window overflows.
- The DoomLoopGate (`src/qwenpaw/loop/gates/doom_loop.py`) works — but only for **identical**
calls: `similarity_threshold=1.0` with `sim = 1-(unique-1)/(total-1)`. Any argument variation
(e.g. a weak model re-reading a truncated file with different `start_line` each round — the
natural truncation-recovery behavior) keeps sim ≤ 0.5 and the gate never fires.
- **Reproduced**: an OpenAI-compatible mock that always returns a `read_file` tool_call with a
*varying* `start_line` ran to `max_iterations=100` (101 model calls, 11s, zero gate events).
With *identical* args the gate stops it at 4 (verified).

## Path B — event stream under degraded infra (fast→medium)

Frontend chat stream path: `POST /console/chat → task_tracker._producer → channel.stream_one`.
Confirmed defects:

1. `run.buffer` is an **unbounded replay buffer** — every SSE string of the run stays in RAM
(`task_tracker.py`, `buffer.append` in `_producer`).
2. Every reconnect **copies the whole buffer** into a new queue (`attach()` /
`attach_or_start()`), and subscriber queues have no `maxsize`.
3. **No stall watchdog**: the heartbeat (`runtime/heartbeat.py`) is a pacemaker (tick + keep
waiting), not a watchdog. A hung upstream holds the run forever.
4. **Failures are silent**: `channel.py` `stream_one`'s generic `except Exception` logs but
emits **no error SSE** (the quota branch does emit one) — the client cannot distinguish
failure from completion and retries the full context. We observed provider connection
errors end the stream silently in 0.2s.
5. Runs **continue after client disconnect** by design; with a slow-but-flowing upstream
(trickle) they effectively never end — read timeouts never fire while chunks keep arriving
(verified: runs alive for tens of minutes after clients were killed).
6. Under heavy parallel tool-call streams we saw repeated
`RuntimeError: async generator ignored GeneratorExit` (SSE generator teardown).

## Path C — keep-alive instance stacking (medium→slow)

`multi_agent_manager` keeps the **old workspace instance alive for up to 24h** when a reload
finds active tasks (`_OLD_WORKSPACE_TASK_MAX_WAIT_ROUNDS`), and the final-resort
`stop(final=False)` **never cancels** the stuck tasks, so the instance's memory is never
released.

- **Reproduced**: with trickle-stalled runs active, `PUT /api/agents/default` (model switch)
logs `Old workspace instance has 4 active task(s)... Scheduling delayed cleanup` +
`Tasks are still active ... Keeping it alive` (60s cadence).
- "Fatten-then-trap" cycles (run a screenshot-heavy task ~3min → switch profile to trickle so
the run never finishes → reload): anonymous memory climbed **monotonically**
503 → 524 → 540 → 544 MB across 3 cycles and never returned. In production with heavier
sessions this is the "switch model and memory never drops" effect from #7222's comments.

## Production signature — and a note on our surrogate reproduction

To be clear: the experimental reproduction below used **surrogate scenarios** (mock model +
fault-injecting proxy + a real DeepSeek-flash) that approximate the real production triggers,
not a reproduction on production infra. Two caveats:

1. **Each factor alone is bounded** (we verified each plateau): long-lived runs (trickle-type
slowness, or approval gates — `rm -rf`-style governed commands park a run awaiting human
approval), fat instances (screenshot review loops are the most effective context filler:
145 `view_image` calls grew the request body 56KB→419KB, right at the 105K-token compaction
threshold), and keep-alive stacking. Only their combination approaches the production
signature.
2. **Strong models resist full triggering**: with DeepSeek-v4-flash, the model chunked its
reads to avoid truncation, and its natural repetition was caught by the DoomLoopGate —
even artificially combining the factors only *approached*, never fully reproduced, ~1MB/s.
The complete, stable reproduction is the real weak-infra recipe in the summary
(vLLM on weak infra + Qwen3.6-27B + the 16× window mismatch): weaker models fall into
varied-args exploration loops more readily, and weak-infra slowness/queueing naturally
produces long-lived runs — together they are the full source of the production signature.

## Reproduction toolkit (available on request)

Single-file, dependency-light: an OpenAI-compatible **loop mock** (modes: identical / varied /
parallel calls) and a **fault-injecting proxy** (`normal | slow_ttft:N | trickle:N | stall |
drop | hang`, hot-switchable via a profile file), plus a chat load driver and dual-side memory
sampler. Happy to open a PR adding them under a `repro/` or `tools/` directory if useful.

## Suggested fixes (small diffs, high value)

1. **DoomLoopGate**: lower default `similarity_threshold` (e.g. 0.7) and/or add a
"same tool N consecutive calls, args ignored" coarse layer. Directly closes Path A.
2. **Keep-alive**: cancel snapshot tasks in the 24h fallback before `stop()`; make the cap
configurable (default much lower). Closes Path C.
3. **Error visibility**: emit an error SSE in `stream_one`'s generic exception branch (mirror
the quota branch). Enables clients to stop retrying blind (Path B amplifier).
4. **Bounded buffers**: `run.buffer` as a bounded deque (replay degrades to "last N events");
subscriber queues with `maxsize` + drop-oldest.
5. **Stall watchdog**: terminate + error SSE after N consecutive heartbeat-only intervals.
6. **Context-length errors**: classify and retry once **after compaction** instead of resending
the same oversized payload (`model_factory` retry path).
7. Optional: `WITH_DESKTOP` build-arg to make the xfce4/Xvfb stack opt-in (webUI deployments
don't use it; ~150–300MB baseline saving). Kept as an *option* — some users like the
agent-OS desktop.

## Environment

Official image `agentscope/qwenpaw:latest` (v2.2.0, `io.qwenpaw.managed-runtime-boundary=2.2.0`),
2 CPUs / 4GB / no swap; model: `deepseek-v4-flash` via local fault-injecting proxy; default
agent config unless stated.

---

**Tooling disclosure (per #4333):** this report was prepared with AI assistance under human direction. All reproduction data comes from real runs (the controlled rig described above plus the reporter's production vLLM deployment); code anchors were verified against the current tree, and the fix measurements in the comment below are from actual test runs. Happy to provide raw logs on request.