agentscope-ai / agentscope-ai/QwenPaw
[Bug]: Empty assistant output_text blocks persisted in session history poison every subsequent request — Ark Responses API returns 400 "MissingParameter: input.content.text"
- Lingua principale
- Python
- Stelle
- 34.9k
- Fork
- 3.1k
- Merge medio
- 1g 15h
- PR unite (30g)
- 225
Descrizione
## QwenPaw Version
v2.1.0 (Docker image based on upstream v2.1.0)
## Description
When using a Volcengine Ark provider (Responses API wire format), a single assistant message with an **empty text block** (`content=[{"type":"output_text","text":""}]`) saved into session history causes the upstream API to reject **every subsequent request** with `400 MissingParameter: input.content.text`. Because QwenPaw replays the full conversation history on each turn, the session is permanently poisoned: the user only sees `Internal error` on every message until the history is manually cleaned.
**How the empty block gets there:** a streaming response that fails / returns empty output gets persisted as an assistant message with an empty `output_text` block (related: #6601, empty responses are not surfaced as errors). On the next turn the poisoned block is replayed upstream verbatim.
**Evidence from production:** a failure-capture hook (logs the full request payload on upstream error) caught a real 953-item / 16 MB session payload that failed with this error. Structural scanning showed the payload was fully valid — 267 correctly paired `function_call`/`function_call_output` items, no orphans, no empty `input_text`, no broken images — and the **only** anomaly was one assistant item with `text: ""`.
**Replay proof:** replaying that exact payload returns the same 400 in <1 s (input-validation stage, no generation). Binary-searching the item list isolated the single empty-text assistant item; replacing `""` with any non-empty string makes the same request return 200.
**Suggested fix:** filter or defend against empty assistant text blocks — either don't persist an assistant turn whose text is empty, or sanitize (drop / placeholder-fill empty `output_text` items) when serializing history for the provider.
**Related PR(s):** N/A
**Security considerations:** N/A (no credentials or secrets involved; upstream request IDs in logs are non-sensitive)
## Component(s) Affected
- [x] Core / Backend (app, agents, config, providers, utils, local_models)
- [ ] Console (frontend web UI)
- [ ] Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
- [ ] Skills
- [ ] CLI
- [ ] Documentation (website)
- [ ] Tests
- [ ] CI/CD
- [ ] Scripts / Deploy
## Environment
- **QwenPaw version:** v2.1.0
- **OS:** Linux (Docker container)
- **Install method:** Docker
- **Python version (if applicable):** 3.11
## Steps to Reproduce
1. Use a provider that speaks the Responses API wire format to Volcengine Ark (doubao/glm plan endpoint).
2. Get an empty assistant text block into session history. Minimal direct repro against the endpoint:
```json
{
"model": "",
"input": [
{"role": "user", "content": [{"type": "input_text", "text": "hi"}]},
{"role": "assistant", "content": [{"type": "output_text", "text": ""}]},
{"role": "user", "content": [{"type": "input_text", "text": "continue"}]}
]
}
```
3. Send the request. It returns:
```
400 {"error":{"code":"MissingParameter","message":"The request failed because it is missing `input.content.text` parameter.","param":"input.content.text","type":"BadRequest"}}
```
4. Replace the empty `""` with any non-empty string (e.g. `"ok"`) → the same request returns `200`.
**Repro matrix (all verified by replaying against the live endpoint):**
| input shape | result |
|---|---|
| `[..., assistant(output_text "")]` anywhere in history (mid-sequence) | **400** |
| same payload with `""` → `"ok"` | 200 |
| `assistant(output_text "")` as the last item | 400 |
| 953-item real poisoned session (one empty block) | 400, <1 s |
| same session truncated to first 941 items (drops the tail containing the empty block) | **200** — removing the block restores the session |
| fully valid large session, no empty blocks | 200 |
## Actual vs Expected
- **Actual:** every subsequent turn in the session fails with upstream 400; QwenPaw surfaces a generic `Internal error` with no hint about the poisoned history item. The empty block is never reported at persistence time.
- **Expected:** QwenPaw should not persist empty assistant text blocks (or should sanitize them before sending), so a single failed/empty response cannot permanently poison the session.
## Logs / Screenshots
Upstream error captured by the gateway failure hook on the real poisoned session:
```
litellm.BadRequestError: OpenAIException - {"error":{"code":"MissingParameter","message":"The request failed because it is missing `input.content.text` parameter. Request id: 0217880026213...","param":"input.content.text","type":"BadRequest"}}
```
The single anomalous item inside the 953-item payload (everything else structurally valid):
```json
{"role": "assistant", "content": [{"type": "output_text", "text": ""}]}
```
Replay of the full 8.3 MB sanitized payload: `HTTP 400 in 0.8s` (input validation, deterministic).
## Additional Notes
- The empty `output_text` block is rejected regardless of position: mid-sequence or trailing. This makes it distinct from model-side flakiness — it is a deterministic input-validation rejection.
- Side observation (low priority, normal traffic unaffected): the same upstream also rejects any input sequence whose **last item is an assistant message** (appears to be a prefill-mode validation). QwenPaw normally ends input with user/tool-result items, so this only matters for history-truncation edge cases.
- Related issues: #5856 (structured tool_call lost during compaction → 400 family), #6601 (empty responses are silently swallowed — likely how the empty block entered history in the first place).
- A gateway-side fallback would only mask the symptom: any fallback model would receive the same invalid empty-text block. The fix belongs in history persistence / outbound serialization.
Update — root cause chain fully traced (how the empty block gets persisted):
We traced the poisoned session end-to-end via QwenPaw logs. The empty assistant block comes from a model turn that completed with zero text output, which QwenPaw then persists unconditionally:
19:19:12 — user sends a message; QwenPaw spends ~3 min on scroll/memory prep, then calls the model.
19:23:29 — the agent turn "finishes" with no text, no tool calls. QwenPaw still saves it as an assistant message (empty output_text) into the persisted session state (session.py "Saved session state").
19:23:32 — usage log confirms the model did respond: prompt_tokens: 45419, completion_tokens: 243 — the 243 completion tokens were all reasoning, with zero text content. So this is not a crashed/aborted stream; it's a successful response whose text part is empty (observed on glm-5.3-flash, reasoning-style model).
19:23:39 — user sends a follow-up; the full-history replay now contains the empty block → Ark 400 MissingParameter on the first call.
19:23:42 — WARNING _agent:_call_model:2431 - Model glm-5.3-flash exhausted all 1 attempt(s) — the BadRequest gets a single attempt, no retry, no usable fallback → surfaced to the user as Internal error.
So the failure requires two cooperating defects: the model occasionally returns a reasoning-only / zero-text response (unavoidable), and QwenPaw persists that empty assistant turn as-is instead of dropping it or raising (the fixable part — this is what #6601 describes from the "empty response is not surfaced" angle).
Side note: in our case the session later self-healed accidentally — the next turn triggered a scroll compaction, which archived the whole poisoned history including the empty block. Sessions without a compaction trigger would stay broken indefinitely, so persistence-time sanitation is still the right fix.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.