agentscope-ai / agentscope-ai/QwenPaw

agent.json systematic corruption: BOM, missing quotes, double-encoding

Abierto
#6,520 3 comentarios 0 reacciones 1 asignado Asignado a @rayrayraykk Ver en GitHub
Lenguaje dominante
Python
Estrellas
34.9k
Forks
3.1k
Merge medio
1 d 15 h
PR fusionados (30 d)
225

Descripción

## Summary

`agent.json` suffered **systemic, distributed corruption** across ~20+ fields, causing complete system failure. Three damage types found: BOM header, missing closing quotes in string values, and double-encoded Chinese text.

## Environment

- **QwenPaw version**: 2.0.1
- **OS**: Windows 10 (AMD64)
- **Agent**: `default` (id=`default`, name=`药老(YaoLao)`)
- **First failure**: 2026-07-27 16:18 CST

## Symptoms

When user tried to switch models in the UI:

1. **Error 1**: `Failed to save active model to agent config`
2. **Error 2** (in logs): `Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)`
3. **System crash**: runtime.py:171 repeated unhandled errors, workspace.py:504 agent failed to start

## Timeline (from qwenpaw.log)

| Time | Event |
|------|-------|
| 12:07 | `config.1cb6c90e.channel-display-migrate.bak` created (clean, valid JSON) |
| 12:13 | `agent.a65ca840.channel-display-migrate.bak` created (clean, valid JSON, 23,266 B) |
| 16:18 | **First BOM error** at `runtime.py:171` — system crash onset |
| 16:25 | `providers.py:675` Failed to get agent-specific model (BOM) |
| 16:25 | `providers.py:760` Failed to save active model (BOM) |
| 16:26 | `workspace.py:504` Failed to start agent instance (BOM) |
| (next day) | User found QwenPaw broken, used external tool to diagnose and fix |

## Forensic Evidence

### File size comparison

| File | Size | BOM? | JSON valid? |
|------|------|------|-------------|
| Migration backup (`agent.a65ca840.*`) | 23,266 B | ❌ No | ✅ Yes |
| Config migration backup (`config.1cb6c90e.*`) | 26,451 B | ❌ No | ✅ Yes |
| Corrupted file (backup) | 26,549 B | ✅ **Yes** (EF BB BF) | ❌ **No** |
| Current (rebuilt) | 24,811 B | ❌ No | ✅ Yes |

### Damage types found in corrupted file

1. **UTF-8 BOM** at position 0 (`EF BB BF`) — Python's `json.loads` throws immediately
2. **~20 `icon` fields** missing closing `"` — string terminator lost, subsequent structure corrupted
3. **`description` field** missing closing `"` — with literal newlines inside value
4. **~126 embedded control characters** (bare CR/LF inside JSON string values)
5. **Double-encoding corruption**: Chinese text shows as garbage (e.g., `药老` → `鑽�鑰?`), classic UTF-8 → GBK → UTF-8 roundtrip damage
6. **8 private-use area characters** (U+E000–U+F8FF range) — partially irreversible

### Critical finding: migration was NOT the cause

The `channel-display-migrate` backup is clean, valid JSON. The corrupted file shares **only 34 bytes** with the clean backup (after skipping BOM), proving the file was **completely rewritten** after migration, not incrementally damaged.

## Root Cause Analysis (Prioritized)

### P0 (Most likely): Missing encoding parameter

When writing `agent.json`, `open(file, 'w')` is called **without specifying `encoding='utf-8'`**. On Windows, the default encoding is CP936 (GBK), which causes:

- BOM bytes (`EF BB BF`) when Python writes UTF-8 data through the CP936 codec
- Chinese characters get double-encoded: UTF-8 bytes → decoded as CP936 → re-encoded as UTF-8

**Check needed**: grep all code paths that write `agent.json` for `open()` calls.

### P1 (High likelihood): String concatenation instead of json.dumps()

Multi-line string values (`description`, `icon`) are being inserted via **string formatting** rather than `json.dumps()`, leading to:

- Bare control characters (LF/CR) inside JSON string values
- Missing closing quotes when the value contains characters that break the delimiter

**Check needed**: Trace how `icon` and `description` fields are serialized.

### P2 (Trigger condition): channel-display-migrate path

The 2.0.1 version upgrade's `channel-display-migrate` likely changed the code path that writes agent.json, exposing the latent P0/P1 defects. Other workspaces (writer-agent, review-agent, chronicle-agent, QA agent) were NOT affected — only `default` was corrupted.

**Check needed**: What's different about the `default` agent's write path vs other agents?

### P3 (Accumulation): Incremental damage over multiple writes

The corruption likely accumulated over several save cycles. Each write added a little more damage until the file became unparseable.

## Recommended Fix

1. **Write encoding**: Use `json.dump(data, f, ensure_ascii=False, indent=2)` with `open(file, 'w', encoding='utf-8')` — **no BOM**
2. **Read encoding**: Use `encoding='utf-8-sig'` to tolerate legacy BOM files
3. **Post-write self-validation**: After writing, immediately `json.loads()` to verify integrity; rollback to previous version on failure
4. **Backup rotation**: Keep last 3 valid versions (`.bak.1`, `.bak.2`, `.bak.3`)
5. **Pre-write guard**: Before saving, validate current file is parseable JSON — reject writes if input is already corrupted

## Evidence Files

All located under `C:\Users\selfe\.qwenpaw\`:

| Item | Path |
|------|------|
| Corrupted backup (raw) | `workspaces/default/agent.json.corrupted_bak` |
| Earlier backup | `workspaces/default/agent.json.bak.20260728100940` |
| Clean migration backup | `workspaces/default/agent.a65ca840.channel-display-migrate.bak` |
| Config migration backup | `config.1cb6c90e.channel-display-migrate.bak` |
| Log evidence | `qwenpaw.log` (search `BOM`, `providers.py:760`, `workspace.py:504`) |
| Full analysis report | `workspaces/default/agent_json_corruption_analysis.md` |

---

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.