agentscope-ai / agentscope-ai/QwenPaw

daily_paper `write_atomic` crashes on PDF with surrogate characters (U+D800–U+DFFF)

クローズ
#7,199 コメント 2 件 リアクション 0 件 担当者 1 名 @jinliyl が担当を希望しています GitHub で見る
主要言語
TypeScript
スター
35k
フォーク
3.1k
平均マージ
1日 13時間
マージ済み PR(30日)
228

説明

## Bug Description

`daily_paper` job crashes when analyzing a paper whose PDF text contains surrogate characters (U+D800–U+DFFF). The crash happens in `write_atomic` -> `content.encode("utf-8")` raises `UnicodeEncodeError: surrogates not allowed`.

The entire job exits with error, and any remaining papers in the daily selection are never processed.

## Environment

- QwenPaw: 2.1.0
- ReMe: 0.4.1.5 (bundled)
- Python: 3.12
- OS: Windows 10

## Root Cause

1. Some PDFs contain text that pypdf extracts as surrogate characters (invalid Unicode)
2. The LLM receives the PDF text in its prompt and may return surrogate characters in its response
3. `write_atomic` at `reme/steps/cookbook/daily_paper/_common.py:97` calls `content.encode("utf-8")` without sanitizing surrogate characters
4. Python's UTF-8 encoder rejects surrogates by design (`UnicodeEncodeError: 'utf-8' codec can't encode characters ... surrogates not allowed`)

## Impact

- Entire daily_paper job fails on a single bad paper
- Remaining selected papers are never processed
- No retry mechanism for individual paper failures

## Suggested Fix

In `_common.py`, the `write_atomic` function should sanitize surrogate characters before encoding:

```python
import re

_SANITIZE_SURROGATES = re.compile(r"[\ud800-\udfff]")

async def write_atomic(path: Path, content: str | bytes) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
lock = await get_path_lock(path)
async with lock:
temp_path = path.with_name(f".{path.name}.{uuid4().hex}.tmp")
if isinstance(content, str):
content = _SANITIZE_SURROGATES.sub("", content)
payload = content.encode("utf-8") if isinstance(content, str) else content
# ... rest unchanged
```

Alternatively, the sanitization could be done earlier in the pipeline (e.g., after PDF text extraction or after LLM response), but `write_atomic` is the most defensive place since it's the final encoding step before writing to disk.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start in reme/steps/cookbook/daily_paper/_common.py at write_atomic and inspect the content encoding path. Reproduce the failure with text containing characters in U+D800–U+DFFF, then verify that writing such content no longer raises UnicodeEncodeError and that normal byte content remains supported.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
backend
issue の種類
バグ
難易度
2/5
見積もり時間
1〜3時間
活発さ
活発
明瞭さ
明確に書かれている
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。