QwenPaw Bug Report: Heartbeat cron session feedback loop (duplicate message pile-up)

# QwenPaw Bug Report: Heartbeat cron session feedback loop (duplicate message pile-up)

**Version checked:** QwenPaw 2.0.1 (deployed). Also verified against the latest `main` branch code on 2026-09-06 — see "Verification" below.
**Severity:** High (agent became unresponsive for ~2 hours; requires manual session-file surgery to recover)
**Affected feature:** Heartbeat cron (HEARTBEAT.md)

## AI-assisted disclosure

This report was assembled by an AI assistant that operates in a real QwenPaw deployment. It investigated the failure, inspected the actual session files on the server, and cross-checked the relevant source code (including the latest `main` branch on GitHub). The findings were reviewed and confirmed by the human operator before submission. The corrupt session JSON backup is real and can be provided (sanitized) on request.

## Summary

On `2026-09-06` (Sunday), a scheduled heartbeat for an agent (`default`, QQ channel) fired at 17:00 (Asia/Shanghai). After that single normal trigger, the agent's `main` session accumulated **17 identical heartbeat user-messages** (user/assistant pairs, all carrying the full HEARTBEAT.md query text) with **no real conversation in between**. The agent stayed busy processing this loop for hours, appearing unresponsive ("reaction got slow, needed many pings to get a reply"), until the session file was manually cleaned.

## Verification against latest upstream (checked 2026-09-06)

Before reporting, we confirmed the issue is still present in current upstream code:

- `src/qwenpaw/app/crons/heartbeat.py` on `main` has **no fix** for this path. The only differences vs. 2.0.1 are `run_sync_io` async-wrapping refactors of sync IO and config reads (verified by diff, 2026-09-06); the request builder still appends a new user message to the persistent `main` session (`"session_id": "main"`, line ~230) on every run, with no dedup, no in-flight guard, and no cleanup of failed/stale turns.
- Related upstream work does not cover this path: #7244 (2026-08-24) and #7268 (2026-08-25) fixed **SSE/stream heartbeat** memory growth and spin-on-timeout issues, but they do not touch the cron-heartbeat session path described here.

## Environment

- QwenPaw version: 2.0.1 (deployed); `main` branch verified same behavior
- Agent: `default`, channel: `qq` (NapCat/onebot-style QQ channel)
- Heartbeat config (`agent.json`):
```json
"heartbeat": {
"enabled": true,
"every": "0 1,5,9,13 * * *", // UTC hours == 09:00/13:00/17:00/21:00 Asia/Shanghai
"target": "last",
"timeout_seconds": 120,
"active_hours": { "start": "08:00", "end": "23:59" }
}
```
- OS: Linux, server timezone UTC (user timezone Asia/Shanghai)

## Bug: Heartbeat messages pile up without dedup / in-flight guard

### Symptom
Session `console/main.json` contained 35 messages: 1 memory header + **17 identical heartbeat request/response pairs** (index 1..34), e.g.

```
[0] role=user name=memory [context compressed]...
[1] role=user name=user "现在是心跳时间。先用 get_current_time ..." (full HEARTBEAT.md)
[2] role=assistant name=团子 thinking/tool_call/...
[3] role=user name=user (identical heartbeat text again)
[4] role=assistant name=团子 ...
... repeated 17 times
```

Nothing else. The messages are byte-identical copies of the heartbeat query. A single cron trigger at 17:00 became 17 injected turns.

### Evidence
- Backup of the corrupt session: `default/sessions/console/main.json.bak_20260906_123203` (35 messages, 17 duplicate heartbeat pairs)
- The same failure pattern happened earlier: `/tmp/qwenpaw_query_error_*.json` dumps from the previous Sunday (`Aug 31`), e.g. `qwenpaw_query_error_81hzhi1h.json` etc.

### Root-cause pointers (code)
- `qwenpaw/app/crons/heartbeat.py` — `run_heartbeat_once()` builds a fresh user request every run:
```python
req = {
"input": [{"role": "user", "content": [{"type": "text", "text": query_text}]}],
"session_id": "main",
...
}
```
Every execution **appends a new user message** to the persistent `main` session. There is **no dedup, no in-flight lock, and no cleanup of stale/aborted heartbeat turns**.
- `qwenpaw/app/crons/manager.py`:
- `HEARTBEAT_MISFIRE_GRACE_SECONDS = 60` — if an execution runs longer than the cron interval tolerance, the next trigger/misfire re-fires and appends again.
- `_heartbeat_callback()` catches exceptions (`logger.exception("heartbeat run failed")`) but never rolls back or removes the failed heartbeat message from the session.
- `timeout_seconds: 120` — once the session is large, the LLM call can't finish within 120s; the run is cancelled, the half-written turn stays, and the next heartbeat compounds the pile.

### Impact
- Agent becomes effectively unresponsive (every new user query is interleaved with 17 duplicate heartbeat tasks).
- Manual surgery of the session JSON is required to recover (we had to strip the session back to the memory header).
- The pile-up keeps growing: more messages → larger context → slower processing → more timeouts → even more pile-up. A positive feedback loop with no upper bound.

### Suggested fix
> **Important design note:** the operator *intentionally* runs heartbeats on the live `main` session so each heartbeat continues the ongoing conversation context (the agent greets the user, per HEARTBEAT.md rules, and needs the chat history). Therefore, fixes must **not** move heartbeats to a separate session by default; they must keep context continuity while stopping duplicates/zombie turns.

1. Add an in-flight guard: skip a heartbeat if the previous one hasn't finished (or an idempotency key per heartbeat fire).
2. On timeout/failure, **remove the just-inserted heartbeat user message** from the session (or mark it failed) instead of leaving a zombie turn. This preserves context continuity while cleaning up the failed turn.
3. Cap the number of consecutive identical heartbeat messages / cap heartbeat messages per session; refuse to append another identical heartbeat if the last N are the same.
4. Optional (opt-in) mitigation: a *separate* volatile "heartbeat scratch" session could be used *only* when context is not needed — but the default behavior with `target=last`/main must keep sharing history, per the operator's design.

## No guard rail against the heartbeat feedback loop

There is no protection anywhere in the heartbeat path: duplicate appends compound each other (more messages → larger context → slower processing → timeouts → more duplicates), and there is no max-consecutive-identical-messages check, no automatic cleanup of zombie heartbeat turns, and no escalation/alert to the user when heartbeats start piling up. (#7244/#7268 addressed the SSE heartbeat path but not this cron-session path.)

## Reproduction steps (best-effort)

1. Enable heartbeat with a cron expression (e.g. `0 1,5,9,13 * * *`), `target: "last"`, on an agent where the last channel is a long-lived QQ session.
2. Let HEARTBEAT.md produce a non-trivial query (our file was ~600 chars with "iron rules").
3. Run the agent's main session until context is large enough that a heartbeat execution exceeds `timeout_seconds` (e.g. slow LLM, big context).
4. Observe: repeated heartbeat user messages accumulate in `main` session; agent replies slow down and eventually stall. Check `console/main.json` for N identical heartbeat pairs.

## Workarounds used on our side

- Manually rewrote `console/main.json` to keep only the memory header, removing the 17 duplicate heartbeat pairs.
- File backed up before editing.

## Contact / context
- Environment: QwenPaw 2.0.1, agent `default`, QQ channel, heartbeat `0 1,5,9,13 * * *` (UTC) with `timeout_seconds: 120`; also verified against `main` branch on 2026-09-06.
- Happy to provide the corrupt session JSON (sanitized) or more logs on request.