agentscope-ai / agentscope-ai/QwenPaw

[Bug]: Empty assistant output_text blocks persisted in session history poison every subsequent request — Ark Responses API returns 400 "MissingParameter: input.content.text"

Aperta
#7,402 3 commenti 0 reazioni 1 assegnatario Rivendicata da @qbc2016 Vedi su GitHub
bug
Lingua principale
Python
Stelle
34.9k
Fork
3.1k
Merge medio
1g 15h
PR unite (30g)
225

Descrizione

## QwenPaw Version

v2.1.0 (Docker image based on upstream v2.1.0)

## Description

When using a Volcengine Ark provider (Responses API wire format), a single assistant message with an **empty text block** (`content=[{"type":"output_text","text":""}]`) saved into session history causes the upstream API to reject **every subsequent request** with `400 MissingParameter: input.content.text`. Because QwenPaw replays the full conversation history on each turn, the session is permanently poisoned: the user only sees `Internal error` on every message until the history is manually cleaned.

**How the empty block gets there:** a streaming response that fails / returns empty output gets persisted as an assistant message with an empty `output_text` block (related: #6601, empty responses are not surfaced as errors). On the next turn the poisoned block is replayed upstream verbatim.

**Evidence from production:** a failure-capture hook (logs the full request payload on upstream error) caught a real 953-item / 16 MB session payload that failed with this error. Structural scanning showed the payload was fully valid — 267 correctly paired `function_call`/`function_call_output` items, no orphans, no empty `input_text`, no broken images — and the **only** anomaly was one assistant item with `text: ""`.

**Replay proof:** replaying that exact payload returns the same 400 in <1 s (input-validation stage, no generation). Binary-searching the item list isolated the single empty-text assistant item; replacing `""` with any non-empty string makes the same request return 200.

**Suggested fix:** filter or defend against empty assistant text blocks — either don't persist an assistant turn whose text is empty, or sanitize (drop / placeholder-fill empty `output_text` items) when serializing history for the provider.

**Related PR(s):** N/A

**Security considerations:** N/A (no credentials or secrets involved; upstream request IDs in logs are non-sensitive)

## Component(s) Affected

- [x] Core / Backend (app, agents, config, providers, utils, local_models)
- [ ] Console (frontend web UI)
- [ ] Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
- [ ] Skills
- [ ] CLI
- [ ] Documentation (website)
- [ ] Tests
- [ ] CI/CD
- [ ] Scripts / Deploy

## Environment

- **QwenPaw version:** v2.1.0
- **OS:** Linux (Docker container)
- **Install method:** Docker
- **Python version (if applicable):** 3.11

## Steps to Reproduce

1. Use a provider that speaks the Responses API wire format to Volcengine Ark (doubao/glm plan endpoint).
2. Get an empty assistant text block into session history. Minimal direct repro against the endpoint:

```json
{
"model": "",
"input": [
{"role": "user", "content": [{"type": "input_text", "text": "hi"}]},
{"role": "assistant", "content": [{"type": "output_text", "text": ""}]},
{"role": "user", "content": [{"type": "input_text", "text": "continue"}]}
]
}
```

3. Send the request. It returns:

```
400 {"error":{"code":"MissingParameter","message":"The request failed because it is missing `input.content.text` parameter.","param":"input.content.text","type":"BadRequest"}}
```

4. Replace the empty `""` with any non-empty string (e.g. `"ok"`) → the same request returns `200`.

**Repro matrix (all verified by replaying against the live endpoint):**

| input shape | result |
|---|---|
| `[..., assistant(output_text "")]` anywhere in history (mid-sequence) | **400** |
| same payload with `""` → `"ok"` | 200 |
| `assistant(output_text "")` as the last item | 400 |
| 953-item real poisoned session (one empty block) | 400, <1 s |
| same session truncated to first 941 items (drops the tail containing the empty block) | **200** — removing the block restores the session |
| fully valid large session, no empty blocks | 200 |

## Actual vs Expected

- **Actual:** every subsequent turn in the session fails with upstream 400; QwenPaw surfaces a generic `Internal error` with no hint about the poisoned history item. The empty block is never reported at persistence time.
- **Expected:** QwenPaw should not persist empty assistant text blocks (or should sanitize them before sending), so a single failed/empty response cannot permanently poison the session.

## Logs / Screenshots

Upstream error captured by the gateway failure hook on the real poisoned session:

```
litellm.BadRequestError: OpenAIException - {"error":{"code":"MissingParameter","message":"The request failed because it is missing `input.content.text` parameter. Request id: 0217880026213...","param":"input.content.text","type":"BadRequest"}}
```

The single anomalous item inside the 953-item payload (everything else structurally valid):

```json
{"role": "assistant", "content": [{"type": "output_text", "text": ""}]}
```

Replay of the full 8.3 MB sanitized payload: `HTTP 400 in 0.8s` (input validation, deterministic).

## Additional Notes

- The empty `output_text` block is rejected regardless of position: mid-sequence or trailing. This makes it distinct from model-side flakiness — it is a deterministic input-validation rejection.
- Side observation (low priority, normal traffic unaffected): the same upstream also rejects any input sequence whose **last item is an assistant message** (appears to be a prefill-mode validation). QwenPaw normally ends input with user/tool-result items, so this only matters for history-truncation edge cases.
- Related issues: #5856 (structured tool_call lost during compaction → 400 family), #6601 (empty responses are silently swallowed — likely how the empty block entered history in the first place).
- A gateway-side fallback would only mask the symptom: any fallback model would receive the same invalid empty-text block. The fix belongs in history persistence / outbound serialization.

Update — root cause chain fully traced (how the empty block gets persisted):

We traced the poisoned session end-to-end via QwenPaw logs. The empty assistant block comes from a model turn that completed with zero text output, which QwenPaw then persists unconditionally:
19:19:12 — user sends a message; QwenPaw spends ~3 min on scroll/memory prep, then calls the model.
19:23:29 — the agent turn "finishes" with no text, no tool calls. QwenPaw still saves it as an assistant message (empty output_text) into the persisted session state (session.py "Saved session state").
19:23:32 — usage log confirms the model did respond: prompt_tokens: 45419, completion_tokens: 243 — the 243 completion tokens were all reasoning, with zero text content. So this is not a crashed/aborted stream; it's a successful response whose text part is empty (observed on glm-5.3-flash, reasoning-style model).
19:23:39 — user sends a follow-up; the full-history replay now contains the empty block → Ark 400 MissingParameter on the first call.
19:23:42 — WARNING _agent:_call_model:2431 - Model glm-5.3-flash exhausted all 1 attempt(s) — the BadRequest gets a single attempt, no retry, no usable fallback → surfaced to the user as Internal error.
So the failure requires two cooperating defects: the model occasionally returns a reasoning-only / zero-text response (unavoidable), and QwenPaw persists that empty assistant turn as-is instead of dropping it or raising (the fixable part — this is what #6601 describes from the "empty response is not surfaced" angle).

Side note: in our case the session later self-healed accidentally — the next turn triggered a scroll compaction, which archived the whole poisoned history including the empty block. Sessions without a compaction trigger would stay broken indefinitely, so persistence-time sanitation is still the right fix.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.