agentscope-ai / agentscope-ai/QwenPaw
agent.json systematic corruption: BOM, missing quotes, double-encoding
- Lenguaje dominante
- Python
- Estrellas
- 34.9k
- Forks
- 3.1k
- Merge medio
- 1 d 15 h
- PR fusionados (30 d)
- 225
Descripción
## Summary
`agent.json` suffered **systemic, distributed corruption** across ~20+ fields, causing complete system failure. Three damage types found: BOM header, missing closing quotes in string values, and double-encoded Chinese text.
## Environment
- **QwenPaw version**: 2.0.1
- **OS**: Windows 10 (AMD64)
- **Agent**: `default` (id=`default`, name=`药老(YaoLao)`)
- **First failure**: 2026-07-27 16:18 CST
## Symptoms
When user tried to switch models in the UI:
1. **Error 1**: `Failed to save active model to agent config`
2. **Error 2** (in logs): `Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)`
3. **System crash**: runtime.py:171 repeated unhandled errors, workspace.py:504 agent failed to start
## Timeline (from qwenpaw.log)
| Time | Event |
|------|-------|
| 12:07 | `config.1cb6c90e.channel-display-migrate.bak` created (clean, valid JSON) |
| 12:13 | `agent.a65ca840.channel-display-migrate.bak` created (clean, valid JSON, 23,266 B) |
| 16:18 | **First BOM error** at `runtime.py:171` — system crash onset |
| 16:25 | `providers.py:675` Failed to get agent-specific model (BOM) |
| 16:25 | `providers.py:760` Failed to save active model (BOM) |
| 16:26 | `workspace.py:504` Failed to start agent instance (BOM) |
| (next day) | User found QwenPaw broken, used external tool to diagnose and fix |
## Forensic Evidence
### File size comparison
| File | Size | BOM? | JSON valid? |
|------|------|------|-------------|
| Migration backup (`agent.a65ca840.*`) | 23,266 B | ❌ No | ✅ Yes |
| Config migration backup (`config.1cb6c90e.*`) | 26,451 B | ❌ No | ✅ Yes |
| Corrupted file (backup) | 26,549 B | ✅ **Yes** (EF BB BF) | ❌ **No** |
| Current (rebuilt) | 24,811 B | ❌ No | ✅ Yes |
### Damage types found in corrupted file
1. **UTF-8 BOM** at position 0 (`EF BB BF`) — Python's `json.loads` throws immediately
2. **~20 `icon` fields** missing closing `"` — string terminator lost, subsequent structure corrupted
3. **`description` field** missing closing `"` — with literal newlines inside value
4. **~126 embedded control characters** (bare CR/LF inside JSON string values)
5. **Double-encoding corruption**: Chinese text shows as garbage (e.g., `药老` → `鑽�鑰?`), classic UTF-8 → GBK → UTF-8 roundtrip damage
6. **8 private-use area characters** (U+E000–U+F8FF range) — partially irreversible
### Critical finding: migration was NOT the cause
The `channel-display-migrate` backup is clean, valid JSON. The corrupted file shares **only 34 bytes** with the clean backup (after skipping BOM), proving the file was **completely rewritten** after migration, not incrementally damaged.
## Root Cause Analysis (Prioritized)
### P0 (Most likely): Missing encoding parameter
When writing `agent.json`, `open(file, 'w')` is called **without specifying `encoding='utf-8'`**. On Windows, the default encoding is CP936 (GBK), which causes:
- BOM bytes (`EF BB BF`) when Python writes UTF-8 data through the CP936 codec
- Chinese characters get double-encoded: UTF-8 bytes → decoded as CP936 → re-encoded as UTF-8
**Check needed**: grep all code paths that write `agent.json` for `open()` calls.
### P1 (High likelihood): String concatenation instead of json.dumps()
Multi-line string values (`description`, `icon`) are being inserted via **string formatting** rather than `json.dumps()`, leading to:
- Bare control characters (LF/CR) inside JSON string values
- Missing closing quotes when the value contains characters that break the delimiter
**Check needed**: Trace how `icon` and `description` fields are serialized.
### P2 (Trigger condition): channel-display-migrate path
The 2.0.1 version upgrade's `channel-display-migrate` likely changed the code path that writes agent.json, exposing the latent P0/P1 defects. Other workspaces (writer-agent, review-agent, chronicle-agent, QA agent) were NOT affected — only `default` was corrupted.
**Check needed**: What's different about the `default` agent's write path vs other agents?
### P3 (Accumulation): Incremental damage over multiple writes
The corruption likely accumulated over several save cycles. Each write added a little more damage until the file became unparseable.
## Recommended Fix
1. **Write encoding**: Use `json.dump(data, f, ensure_ascii=False, indent=2)` with `open(file, 'w', encoding='utf-8')` — **no BOM**
2. **Read encoding**: Use `encoding='utf-8-sig'` to tolerate legacy BOM files
3. **Post-write self-validation**: After writing, immediately `json.loads()` to verify integrity; rollback to previous version on failure
4. **Backup rotation**: Keep last 3 valid versions (`.bak.1`, `.bak.2`, `.bak.3`)
5. **Pre-write guard**: Before saving, validate current file is parseable JSON — reject writes if input is already corrupted
## Evidence Files
All located under `C:\Users\selfe\.qwenpaw\`:
| Item | Path |
|------|------|
| Corrupted backup (raw) | `workspaces/default/agent.json.corrupted_bak` |
| Earlier backup | `workspaces/default/agent.json.bak.20260728100940` |
| Clean migration backup | `workspaces/default/agent.a65ca840.channel-display-migrate.bak` |
| Config migration backup | `config.1cb6c90e.channel-display-migrate.bak` |
| Log evidence | `qwenpaw.log` (search `BOM`, `providers.py:760`, `workspace.py:504`) |
| Full analysis report | `workspaces/default/agent_json_corruption_analysis.md` |
---
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.