AI SDK request serialization masks UTF-8 bytes producing bare control characters -> DeepSeek 400 "control character found"

Title

AI SDK request serialization masks UTF-8 bytes producing bare control characters → DeepSeek 400 "control character found"

Environment

• opencode 1.18.12 (npm opencode-ai@1.18.12, latest as of 2026-08-04)
• OS: Windows 11
• Runtime: Bun 1.3.14
• Provider: DeepSeek via @ai-sdk/openai-compatible, baseURL: http://127.0.0.1:8899/v1 (local transparent capturing proxy), model deepseek-v4-flash

Summary

In sessions with long Chinese-language context, outgoing POST /v1/chat/completions request bodies occasionally contain 100–212 raw control bytes (0x00–0x1F) embedded in JSON string values. The body is then neither valid UTF-8 nor valid JSON, and DeepSeek rejects it with HTTP 400 control character found / expected ',' or '}'. Retrying after compaction (which rebuilds the body) succeeds — so the corruption is re-generated on the serialization path, not stored in the session database.

The corruption is a bit-mask applied to the non-leading bytes of UTF-8 multi-byte sequences:

Original UTF-8 (char)	Corrupted bytes	Masked byte
默 U+9ED8 = E9 BB 98	E9 BB 18	0x98 & 0x1F = 0x18
文 U+6587 = E6 96 87	E6 96 07	0x87 & 0x1F = 0x07
成 U+6210 = E6 88 90	E6 88 10	0x90 & 0x1F = 0x10
识 U+8BC6 = E8 AF 86	E8 AF 06	0x86 & 0x1F = 0x06

All 201 observed corrupt bytes in one captured body fall in 0x00–0x1F (histogram covers 0x00–0x1F uniformly except 0x1B); & 0x7F cannot explain this (it would produce ≈half 0x20–0x3F results). This is consistent with & 0x1F (or an equivalent narrower mask) being applied to bytes 0x80–0xBF.

Notably, the corrupted content includes static system-prompt text (e.g. Windows 默\x18认 cp936 from a user AGENTS.md, corrupt byte at body offset 62142) — so this is not a tool-output-specific issue; any long-enough body with Chinese text is at risk.

Where the corruption happens (evidence chain)

1. A read-only plugin hook (experimental.chat.messages.transform) inspects outgoing messages right before send → ctrl_total=0 (clean).
2. The transparent proxy records the raw HTTP body → 104–212 bare control bytes (dirty).
3. Nothing else runs between the two points except opencode's provider/AI SDK serialization.

→ The corruption is introduced by the serializer between the transform hook and the HTTP write. It cannot be fixed locally; it needs an upstream fix.

Reproduction

1. Configure DeepSeek provider via @ai-sdk/openai-compatible.
2. Have a session whose context (system prompt + conversation) contains substantial Chinese text.
3. Issue several turns; at some point the outgoing body contains bare control bytes in string values.
4. DeepSeek returns 400 {"error":{"message":"... control character found ..."}}.

Heuristic trigger: long bodies with many Chinese characters. The failure is intermittent (compaction of the session rebuilds the body and the next attempt succeeds).

DeepSeek behavior (verified directly)

Payload form	Result
Escaped \u00xx / \" / \u007f inside JSON strings	HTTP 200 (accepted)
Bare control bytes (0x00–0x1F) inside JSON strings	HTTP 400 (rejected)

So the fix is straightforward at the serializer level: string values must be escaped per RFC 8259 §7 (the semantics of JSON.stringify), or at minimum control characters must be escaped/removed before the body is written.

Expected behavior

The request body sent to the provider must be valid JSON: control characters inside string literals must be escaped (\u00xx, \t, \n, \r, ...), and UTF-8 sequences must not be bit-masked.

Suggested direction

• Locate where code points/bytes are bit-masked (& 0x1F-like operation) during provider request construction and remove it.
• Extend the existing outbound sanitization pass (the same channel as sanitizeSurrogates() for lone surrogates, see related issues) to escape control characters 0x00–0x1F/0x7F in string values before sending.

Related (different mechanism, same "request body not RFC 8259-clean" family)

• #13988 / #14630 / #17758 — lone UTF-16 surrogates serialized as \uD8xx escapes rejected by strict parsers (already fixed in sanitize-surrogates.ts, commit 9da720414). Different: that case produces escaped ASCII; this case produces bare control bytes from bit-masking.
• anthropics/claude-code#53463 — unescaped U+0000–U+001F in JSON string fields (serialization escaping defect in the same family).

Evidence files (sanitized)

• Full captured request body (API key redacted): req_20260804_122817_sanitized.bin — 201 bare control bytes; decode fails at offset 61214; first corrupt byte at 62142 (Windows 默\x18认 cp936).
• Capture shows the corrupt bytes live inside messages[*].content strings.

---