zai-org / zai-org/feedback

[Bug] 大输出预览按 UTF-16 码元截断可切碎代理对,孤立代理随会话历史进入服务端致确定性 500,会话永久报废

Open
#689 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority: P2
Dominant language
No language data
Stars
22
Forks
1
PR merge metrics
No merged PRs in 30d

Description

环境

  • ZCode Desktop 3.12.3(3.12.3.7463),macOS Sequoia 15.6(darwin 24.6.0, Apple Silicon)
  • 模型请求走 GLM API(服务端返回 500)

现象

某次工具输出超大(121.6KB,其中含大段 CJK 扩展B区字符,即基本平面以外的字符)后,会话进入永久失败状态:此后每一个模型请求都返回

TerminalStreamChunkError: [500]['utf-8' codec can't encode character '\ud840' in position 955242: surrogates not allowed]

同一会话在约 1 小时内先后三次请求(每次内部重试 11 次)全部以同一错误告终;手动重试与 /compact 均无效(compact 请求同样携带完整历史),会话只能弃用重开。

根因(已在原生证据上逐环验证)

  1. 超大输出触发 Output too large ... Preview (first 2KB) 路径,预览按 UTF-16 码元个数截断(slice(0, 2048) 类逻辑)。基本平面以外字符(emoji、CJK 扩展区汉字等)在 UTF-16 中各占 2 个码元;当截断点恰好落在某个代理对中间时,预览末尾残留一个孤立的高位代理 \ud840
  2. 该预览作为工具结果进入会话历史,此后每个请求的 payload 都携带它。
  3. Node 端(JSON.parse / JSON.stringify,ES2019 well-formed 输出)对孤立代理宽容、仅做转义,客户端全程无任何异常;GLM API 服务端(Python)按 Unicode 规范拒绝 str.encode('utf-8')UnicodeEncodeError → HTTP 500。

原生证据

  • 会话 rollout 文件(~/.zcode/cli/rollout/model-io-<session>.jsonl)中,该工具结果的预览文本以字面转义 \ud840 结尾(其后无低位代理)。严格 JSON 解析器 jq 直接拒绝该行:jq: parse error: Invalid \uXXXX surrogate pair escape,可证明是孤立代理而非合法字符对。
  • 同一输出的完整落盘原件(~/.zcode/cli/exec/<session>/call_*-stdout.log)经逐字符扫描含 0 个孤立代理——坏字符只在「内联预览」截断环节引入,并非源输出自带。
  • 服务端三次报错的 position 完全一致(955242),属确定性失败。

最小机制演示(与具体项目无关)

$ node -e 'const s="前缀𠀀𠀁𠀂"; console.log("可见5字符, s.length =", s.length);
const p=s.slice(0,5); console.log("slice(0,5) =", JSON.stringify(p));'
可见5字符, s.length = 8          ← 每个扩展B区字符占 2 个码元
slice(0,5) = "前缀𠀀\ud840"      ← 切在代理对中间,孤立代理被转义输出

$ printf '{"preview":"\\ud840"}' | python3 -c 'import json,sys
json.loads(sys.stdin.read())["preview"].encode("utf-8")'
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud840'
in position 2: surrogates not allowed    ← 服务端 500 的直接来源

$ printf '{"preview":"\\ud840"}' | jq .
jq: parse error: Invalid \uXXXX surrogate pair escape

期望行为

  1. 截断按码点边界:预览等所有按长度截断用户可见文本的位置,应使用码点安全截断(如 Array.from(str).slice(0, n).join('')),或截断后检测末尾孤立高位代理并剔除。
  2. 请求前兜底清洗:发送模型请求前对 payload 做一次孤立代理检测/清洗,避免单条脏数据永久毒化整个会话。
  3. 不可恢复错误的处理:对确定性载荷类 5xx(同一 position 反复失败),重试策略无法恢复,建议检测到后明确提示用户,并支持跳过污染消息以恢复会话。

影响

单个会话永久不可用,只能弃用重开;会话内未及时落盘的工作有丢失风险。

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository source files or tests are identified. Start by reproducing the UTF-16 boundary case described in the issue, then inspect the rollout file at ~/.zcode/cli/rollout/model-io-.jsonl and the corresponding exec log to trace preview creation and request serialization. Done means the requested safe truncation, payload cleanup, and recovery handling are addressed and the demonstrated failure no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, python
Domain
api, backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.