[Bug] CoPaw runtime Manager 长时间运行后 Matrix 入站消息被 silent drop || [Bug] Matrix inbound messages are silently dropped after CoPaw runtime Manager runs for a long time

## 现象 / Symptoms

在多 Worker（Hermes / OpenClaw / 混合）协作场景下，CoPaw runtime 的 Manager 长时间运行（数十分钟～小时级）后，**会突然停止响应房间里的所有消息**，但进程不挂、没有任何 error/warning 日志：

- Matrix room 里 admin 重新 `@manager:host`、Worker 之间 reply、@ 全 MXID + 显式带 `m.mentions=[manager]`，**全部没反应**；
- typing indicator 不出现，read receipt 不更新；
- 容器内 `python3 -m copaw app` 进程仍 alive（`/proc/1/status` = `S (sleeping)`、22 threads，PID 1 在 `epoll_poll`）；
- `~/.copaw/copaw.log` 最后一条是若干分钟前正常处理完一条消息后的 `Saved session state to ...`，**之后再无任何写入**；
- `~/.copaw/matrix_sync_token` 文件持续被改写，长轮询 `/sync` 没死。

`docker restart hiclaw-manager`（K8s 下 `kubectl rollout restart`）后**立刻恢复**，并在启动 sync loop 后回放此前所有被丢的事件——证明事件确实送达了 nio 但 callback 没走完。

跟之前的 #628 「Manager 没有响应」、#627 「manager 很容易失忆了」可能是同一类「manager silent-hang」问题在不同触发条件下的实例。

## 触发条件 / Reproduction Conditions

我在调试 PR #659（hermes worker 集成）时连续两次遇到，触发场景的共同点：

1. **CoPaw runtime 的 Manager**（`HICLAW_MANAGER_RUNTIME=copaw`，镜像 `hiclaw/hiclaw-manager-copaw:latest`）；
2. **房间里有 ≥2 个 Worker**（Hermes 或 OpenClaw 都触发过，但 Hermes 因为 spam 更多，触发更快）；
3. **持续运行至少几十分钟**，房间里有以下任意一类「高频/异常」inbound 事件：
- 旧版 Hermes worker 每次 session 都自动发的 `📬 No home channel is set for Matrix...` reminder（PR #659 已 fix，但混用新旧镜像时仍会出现）；
- Hermes 的 approval gate 触发 `Reply /approve to execute, …` 卡点消息（PR #659 已 fix，需要 `HERMES_YOLO_MODE=1`）；
- Worker 启动时一次性发出的 `✅ Config sync completed!` 汇报（多个 worker 同时启动会同时发到 manager 房间）。

也就是说，**无危害的 housekeeping 类消息也会触发**，不需要恶意/异常输入。

## 复现一次的诊断证据 / Diagnostic Evidence

环境：本地 Podman + 单 manager (`copaw` runtime) + 3 个 Hermes worker（alice / bob / charlie），group room `!gZ3NGbzrweqQJS9f2x` 含 admin + alice + manager。

### 1. `/sync` 长轮询持续推进 + manager 自己的 token 能拉到所有消息

```
$ cat /root/manager-workspace/.copaw/matrix_sync_token
1895
$ curl ".../sync?timeout=0" -H "Authorization: Bearer $MGR_TOKEN"
{"next_batch":"1898", ...} # server 端有 3 条新增没拉

# 用 manager 自己的 access_token 拉房间最近事件，能完整看到所有 worker/admin 消息
$ curl ".../rooms//messages?dir=b&limit=10" -H "Authorization: Bearer $MGR_TOKEN"
… alice / admin / manager 所有消息都在 …
```

### 2. 同房间、同 PUT body、仅换 sender，行为差异巨大

故障期间用 `@admin` 和 `@alice` 的 access_token 直接 PUT `m.room.message`，body / `m.mentions` 完全一致：

```jsonc
// alice 发的 → manager 完全没处理
{"msgtype":"m.text","body":"@manager:… alice test A","m.mentions":{"user_ids":["@manager:matrix-local.hiclaw.io:18080"]}}

// admin 发的 → manager 5 秒内 LLM 回复
{"msgtype":"m.text","body":"@manager:… admin test A","m.mentions":{"user_ids":["@manager:matrix-local.hiclaw.io:18080"]}}
```

`copaw.log` 里只有 admin 那条出现：

```
07:50:21 | INFO | runner/runner.py:404 | Handle agent query:
{
"channel": "matrix",
"msgs_str": "[Msg(... text='admin 💕: @manager:… admin test A')]"
}
```

`alice test A` 既没进 `Handle agent query`，**也没进 history**（admin 之后再发任何消息时 context 里都没出现它），但 `matrix_sync_token` 在 alice 这条之后**确实推进了**。

> 测试中尝试过 reply 关系、formatted_body 含/不含 `matrix.to`、body 含/不含全 MXID、`m.mentions` 含/不含等组合，**只要 sender 是 worker（alice/bob/charlie），manager 全部 silent drop；admin 发同结构永远能进 LLM**。`_check_allowed` 配的 `group_allow_from` / `group_policy=allowlist` / `groups: {*: {requireMention: true}}` 配置一致，理论上不该差异化。

### 3. 重启后没改任何配置就立刻恢复 + 回放被丢的事件

```
2026-04-21 07:57:05 | INFO | matrix/channel.py:506 | restored sync token from /…/matrix_sync_token
2026-04-21 07:57:05 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@bob:… body='📬 No home channel is set …'
2026-04-21 07:57:05 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@charlie:… body='📬 No home channel is set …'
2026-04-21 07:57:06 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@alice:… body='✅ Config sync completed …'
2026-04-21 07:57:06 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@bob:… body='Config sync completed successfully …'
2026-04-21 07:57:30 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@alice:… body='@manager:… alice debug after restart'
```

被丢的事件**并不是 server 没送达**，是**送达后 callback 没走完**。重启后用持久化 token 续上即可。

## 调试时踩的坑 / Debug Friction

`copaw 1.0.2` 自带的 `copaw/app/channels/matrix/channel.py:43` 用的是改名前的旧 logger 名 `qwenpaw.channels.matrix`，但 `copaw/utils/logging.py` 的 `LOG_NAMESPACE = "copaw"` 只给 `copaw.*` 命名空间挂 file handler。结果是 **matrix channel 的所有 `_on_room_event` / `_check_allowed` / `_was_mentioned` / `_is_dm_room` 等 INFO/DEBUG 日志全部被 root 默默丢弃**，调查 hang 的时候完全没法判断 callback 是否触发，必须 `sed -i` patch 后重启 manager 才能拿到证据。

这个 logger 名属于 CoPaw 上游遗留，建议 hiclaw 在 `manager/Dockerfile.copaw` 或启动脚本里对 vendored CoPaw 做一个 sed 修正，或者跟 CoPaw 上游 sync 一下。

## 临时缓解 / Workaround

1. **重启整个 manager 容器**：长轮询会用持久化的 `matrix_sync_token` 续上，被丢的事件会被回放（不会丢消息，但 LLM 会一次性突然回应一堆历史事件）。
2. **减少触发因素**：本次 PR #659 已经做了两件事降低 hermes worker 对 manager 的 spam，建议 OpenClaw worker 也对照检查：
- `HERMES_YOLO_MODE=1` 跳过 dangerous-command approval gate（`Reply /approve to execute …`）；
- `MATRIX_HOME_CHANNEL=disabled` 抑制每次 session 的 `📬 No home channel …` reminder。
3. **多 worker 升级时确认所有 worker 都用了新镜像**：如果只升 alice、bob/charlie 留旧镜像，旧 worker 会继续 spam，照样能把 manager 拖 hang。

## 建议跟进 / Suggested Next Steps（按优先级）

- **[P0] hiclaw 这一侧的 watchdog**：给 manager 容器加一个轻量 health-check（比如 `~/.copaw/copaw.log` 的 `Handle agent query` 时间戳 + 最近 sync_token mtime 双指标），N 分钟没新 query 但 sync 还在动 → 自动重启 manager pod / container。这个能直接缓解线上影响，不依赖上游修。
- **[P0] 文档**：在 manager-copaw runtime 的 troubleshooting 文档里写一段「manager 不响应时先看 copaw.log 是否在 N 分钟前停了，是的话重启」。
- **[P1] 把 logger 名 fix 提到 vendored CoPaw 或上游**：`qwenpaw.channels.matrix` → `copaw.matrix.channel`（一行 sed），让后续诊断不再瞎。
- **[P1] 跟 CoPaw 上游对接**：等 hiclaw 这边再拿一次更精确的 trace（建议用 `py-spy dump --pid 1` 或开 nio TRACE log 复现），再去 [agentscope-ai/QwenPaw](https://github.com/agentscope-ai/QwenPaw) 报上游。我在调试时一度提了 [QwenPaw#3646](https://github.com/agentscope-ai/QwenPaw/issues/3646) 但发现 hiclaw 视角更适合先收拢追踪，已 close。CoPaw 已有一个相似症状的 issue [QwenPaw#3640](https://github.com/agentscope-ai/QwenPaw/issues/3640)（MCP TaskGroup 异常导致 Agent 假死无报错），但根因路径不一样。
- **[P2] 防御性加固**：考虑给 `MatrixChannel._on_room_event` 加超时/异常 wrapper（在 vendored copy 里 patch 也行）；给 `unified_queue_manager.py` 的 per-session consumer 加 health probe，发现 consumer started 但持续不消费时记 ERROR。

## 相关 / Related

- PR #659 — Hermes worker 集成，触发本次复现；该 PR 已加 `HERMES_YOLO_MODE=1` + `MATRIX_HOME_CHANNEL=disabled`，把两类常见 spam 源头堵了。
- #628 — Manager 没有响应（403）—— 网络层另一类不响应。
- #627 — manager 很容易失忆了 —— session 持久化相关，跟本 issue 不同根因但同样表现为「manager 行为异常」。
- QwenPaw#3640 — Agent 假死无报错（MCP TaskGroup），同类「silent-hang」模式在 CoPaw 内的另一处。

## 环境 / Environment

```
hiclaw: feat/hermes-worker-runtime branch (PR #659)
manager image: hiclaw/hiclaw-manager-copaw:latest (HICLAW_MANAGER_RUNTIME=copaw)
worker image: hiclaw/hermes-worker:latest (3 instances: alice/bob/charlie)
copaw: 1.0.2
matrix-nio: 0.25.2
Python: 3.11.15
Platform: Linux aarch64 container (Podman on macOS)
Matrix server: tuwunel/conduwuit fork
Room: group room (3 members), unencrypted, requireMention=true
```

---
## Phenomenon / Symptoms

In a multi-Worker (Hermes/OpenClaw/Hybrid) collaboration scenario, after the CoPaw runtime Manager runs for a long time (tens of minutes to hours), it will suddenly stop responding to all messages in the room, but the process will not hang and there will be no error/warning logs:

- In the Matrix room, the admin re-replicates `@manager:host`, reply between workers, @ full MXID + explicit `m.mentions=[manager]`, **no response**;
- The typing indicator does not appear and the read receipt is not updated;
- The `python3 -m copaw app` process in the container is still alive (`/proc/1/status` = `S (sleeping)`, 22 threads, PID 1 in `epoll_poll`);
- `~/.copaw/copaw.log` The last entry is `Saved session state to...` after a message was processed normally a few minutes ago, **nothing is written after that**;
- The `~/.copaw/matrix_sync_token` file continues to be overwritten, long polling `/sync` is not dead.

**Recover immediately** after `docker restart hiclaw-manager` (`kubectl rollout restart` under K8s), and play back all previously lost events after starting the sync loop - proving that the event was indeed delivered to nio but the callback did not finish.

Like the previous #628 "Manager is not responding" and #627 "Manager easily loses his memory", they may be instances of the same type of "manager silent-hang" problem under different triggering conditions.

## Trigger conditions / Reproduction Conditions

I encountered this twice in a row while debugging PR #659 (hermes worker integration), the common denominator of the triggering scenarios:

1. **Manager** of CoPaw runtime (`HICLAW_MANAGER_RUNTIME=copaw`, mirroring `hiclaw/hiclaw-manager-copaw:latest`);
2. **There are ≥2 Workers in the room** (Hermes or OpenClaw are both triggered, but Hermes triggers faster because it has more spam);
3. **Continuously running for at least dozens of minutes**, there are any of the following "high-frequency/abnormal" inbound events in the room:
- The old version of Hermes worker automatically sends the `📬 No home channel is set for Matrix...` reminder every session (PR #659 has been fixed, but it will still appear when mixing old and new images);
- Hermes' approval gate triggers `Reply /approve to execute, ...` stuck message (PR #659 has been fixed, requires `HERMES_YOLO_MODE=1`);
- A one-time `✅ Config sync completed!` report is sent when the Worker starts (multiple workers started at the same time will be sent to the manager room at the same time).

That is, harmless housekeeping messages will also trigger, no malicious/unusual input is required.

## Diagnostic Evidence that is repeated once / Diagnostic Evidence

Environment: local Podman + single manager (`copaw` runtime) + 3 Hermes workers (alice / bob / charlie), group room `!gZ3NGbzrweqQJS9f2x` contains admin + alice + manager.

### 1. `/sync` long polling continues to advance + the manager’s own token can pull all messages

```
$ cat /root/manager-workspace/.copaw/matrix_sync_token
1895
$ curl ".../sync?timeout=0" -H "Authorization: Bearer $MGR_TOKEN"
{"next_batch":"1898", ...} # There are 3 new items on the server side.

# Use the manager's own access_token to pull recent events in the room, and you can fully see all worker/admin messages
$ curl ".../rooms//messages?dir=b&limit=10" -H "Authorization: Bearer $MGR_TOKEN"
…alice/admin/manager all messages are in…
```

### 2. Same room, same PUT body, only changing sender, huge difference in behavior

During the fault, use the access_token of `@admin` and `@alice` to directly PUT `m.room.message`, and the body / `m.mentions` are exactly the same:

```jsonc
// Sent by alice → manager didn’t handle it at all
{"msgtype":"m.text","body":"@manager:… alice test A","m.mentions":{"user_ids":["@manager:matrix-local.hiclaw.io:18080"]}}

// Sent by admin → manager LLM will reply within 5 seconds
{"msgtype":"m.text","body":"@manager:… admin test A","m.mentions":{"user_ids":["@manager:matrix-local.hiclaw.io:18080"]}}
```

Only the admin entry appears in `copaw.log`:

```
07:50:21 | INFO | runner/runner.py:404 | Handle agent query:
{
"channel": "matrix",
"msgs_str": "[Msg(... text='admin 💕: @manager:... admin test A')]"
}
```

`alice test A` neither entered `Handle agent query` nor history** (it did not appear in the context when admin sent any messages after that), but `matrix_sync_token` did advance after alice's message.

> In the test, we tried the reply relationship, formatted_body with/without `matrix.to`, body with/without full MXID, `m.mentions` with/without and other combinations. **As long as the sender is a worker (alice/bob/charlie), the manager will all silently drop; the admin can always enter LLM with the same structure**. The configurations of `group_allow_from` / `group_policy=allowlist` / `groups: {*: {requireMention: true}}` configured with `_check_allowed` are consistent and should not be differentiated in theory.

### 3. Restore immediately after restarting without changing any configuration + play back the lost events

```
2026-04-21 07:57:05 | INFO | matrix/channel.py:506 | restored sync token from /…/matrix_sync_token
2026-04-21 07:57:05 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@bob:… body='📬 No home channel is set…'
2026-04-21 07:57:05 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@charlie:… body='📬 No home channel is set…'
2026-04-21 07:57:06 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@alice:… body='✅ Config sync completed…'
2026-04-21 07:57:06 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@bob:… body='Config sync completed successfully …'
2026-04-21 07:57:30 | INFO | matrix/channel.py:1479 | _on_room_event: sender=@alice:… body='@manager:… alice debug after restart'
```

The lost event is not that the server has not been delivered, but that the callback has not been completed after being delivered. After restarting, just use the persistence token to renew.

## Pitfalls during debugging / Debug Friction

The `copaw/app/channels/matrix/channel.py:43` that comes with `copaw 1.0.2` uses the old logger name `qwenpaw.channels.matrix` before the name change, but the `LOG_NAMESPACE = "copaw"` of `copaw/utils/logging.py` only hangs the file handler for the `copaw.*` namespace. The result is that all `_on_room_event` / `_check_allowed` / `_was_mentioned` / `_is_dm_room` and other INFO/DEBUG logs of **matrix channel are silently discarded by root**. When investigating the hang, it is completely impossible to judge whether the callback is triggered. You must restart the manager after `sed -i` patch to get the evidence.

This logger name belongs to CoPaw's upstream legacy. It is recommended that hiclaw make a sed correction to vendored CoPaw in `manager/Dockerfile.copaw` or startup script, or sync it with CoPaw's upstream.

## Temporary mitigation / Workaround

1. **Restart the entire manager container**: Long polling will be continued with the persistent `matrix_sync_token`, and the lost events will be played back (no messages will be lost, but LLM will suddenly respond to a bunch of historical events at once).
2. **Reduce trigger factors**: This PR #659 has done two things to reduce the spam of hermes worker to manager. It is recommended that OpenClaw worker also check:
- `HERMES_YOLO_MODE=1` skips dangerous-command approval gate (`Reply /approve to execute…`);
- `MATRIX_HOME_CHANNEL=disabled` suppresses the `📬 No home channel…` reminder for each session.
3. **When upgrading multiple workers, make sure all workers use the new image**: If you only upgrade alice, bob/charlie and keep the old image, the old worker will continue to spam, and the manager can still be hung.

## Suggested follow-up / Suggested Next Steps (by priority)

- **[P0] Watchdog on hiclaw side**: Add a lightweight health-check to the manager container (such as `Handle agent query` timestamp of `~/.copaw/copaw.log` + recent sync_token mtime double indicator), if there is no new query for N minutes but sync is still moving → automatically restart the manager pod/container. This can directly alleviate the online impact and does not rely on upstream repairs.
- **[P0] Document**: Write a paragraph in the troubleshooting document of manager-copaw runtime "When the manager does not respond, first check whether the copaw.log stopped N minutes ago, and if so, restart it."
- **[P1] Mention the logger name fix to vendored CoPaw or upstream**: `qwenpaw.channels.matrix` → `copaw.matrix.channel` (one line of sed), so that subsequent diagnosis will no longer be blind.
- **[P1] Connect with CoPaw upstream**: Wait for hiclaw to get a more accurate trace (it is recommended to use `py-spy dump --pid 1` or open nio TRACE log to reproduce), and then go to [agentscope-ai/QwenPaw](https://github.com/agentscope-ai/QwenPaw) to report to the upstream. I once mentioned [QwenPaw#3646](https://github.com/agentscope-ai/QwenPaw/issues/3646) during debugging, but found that the hiclaw perspective is more suitable for closing the tracking first, and it has been closed. CoPaw already has an issue [QwenPaw#3640](https://github.com/agentscope-ai/QwenPaw/issues/3640) with similar symptoms (MCP TaskGroup exception causes Agent to feign death without reporting an error), but the root cause path is different.
- **[P2] Defensive reinforcement**: Consider adding a timeout/exception wrapper to `MatrixChannel._on_room_event` (patch in vendored copy will also work); add a health probe to the per-session consumer of `unified_queue_manager.py`, and record ERROR when it is found that the consumer started but continues not to consume.

## Related / Related

- PR #659 — Hermes worker integration triggers this recurrence; this PR has added `HERMES_YOLO_MODE=1` + `MATRIX_HOME_CHANNEL=disabled`, blocking two common sources of spam.
- #628 - Manager Not Responding (403) - Another type of unresponsiveness at the network layer.
- #627 — Managers can easily lose their memory — session persistence is related to this issue. It has a different root cause than this issue but also shows "abnormal manager behavior".
- QwenPaw#3640 - Agent fakes death without error (MCP TaskGroup), the same "silent-hang" mode is in another place in CoPaw.

## Environment / Environment

```
hiclaw: feat/hermes-worker-runtime branch (PR #659)
manager image: hiclaw/hiclaw-manager-copaw:latest (HICLAW_MANAGER_RUNTIME=copaw)
worker image: hiclaw/hermes-worker:latest (3 instances: alice/bob/charlie)
copaw: 1.0.2
matrix-nio: 0.25.2
Python: 3.11.15
Platform: Linux aarch64 container (Podman on macOS)
Matrix server: tuwunel/conduwuit fork
Room: group room (3 members), unencrypted, requireMention=true
```