envoyproxy / envoyproxy/envoy

ext_proc FULL_DUPLEX_STREAMED: duplicate RequestBody chunk with EndOfStream=true on large bodies (1.35+)

Open
#46,237 8 comments 0 reactions 1 assignee Claimed by @yanjunxiang-google View on GitHub
area/ext_proc bug
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
428

Description

## Summary

With `request_body_mode: FULL_DUPLEX_STREAMED`, Envoy can re-deliver the final `RequestBody` chunk (with `end_of_stream=true`) after the ext_proc server has already received and processed an `end_of_stream=true` chunk for the same request. This causes the ext_proc server to re-process an already-complete body, leading to parse failures.

## Versions affected

| Istio version | Envoy version | Duplicate EoS observed |
|---|---|---|
| 1.26.2 | 1.34 | No (3/3 pass) |
| 1.27.9 | 1.35.13 | Yes (~2/3 fail) |
| 1.28.8 | 1.36.9 | Yes (~4/5 fail) |
| 1.29.5 | 1.37.5 | Yes (~7/8 fail) |

The failure rate increases with newer Envoy versions, suggesting a regression introduced in 1.35.

## Reproduction

**Setup:** Single ext_proc filter with `request_body_mode: FULL_DUPLEX_STREAMED`, processing JSON inference requests.

**Trigger:** Send a large (~1MB+) request body. Multi-turn chat payloads consistently reproduce this.

```bash
# Generate a ~1.2MB JSON body
python3 -c "
import json
msgs = [{'role':'user','content':'analyze this: ' + 'x'*500}]
for i in range(20):
msgs.append({'role':'assistant','content':[{'type':'output_text','text':'y'*2000}]})
msgs.append({'role':'user','content':'check function ' + str(i)})
json.dump({'model':'gpt-5.5','input':msgs}, open('body.json','w'))
"

curl -d @body.json \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
https:///v1/responses
```

**Expected:** One `RequestBody` message with `end_of_stream=true`, followed by no further `RequestBody` messages for this request.

**Actual:** Two `RequestBody` messages with `end_of_stream=true` for the same stream. Server logs:

```
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":false} x 7
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":true} <- first EoS, processed
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":true} <- duplicate EoS
```

**Same payload without ext_proc:** 100% success (50/50 at 2.5MB), confirming the issue is in ext_proc delivery, not upstream.

## Impact

Any ext_proc server that accumulates body chunks and processes on `end_of_stream=true` will re-process the already-complete buffer with the duplicate chunk appended, causing parse failures. This affects real workloads -- multi-turn LLM conversations routinely exceed 1MB.

We have worked around this in [llm-d-inference-payload-processor#225](https://github.com/llm-d/llm-d-inference-payload-processor/pull/225) by tracking body completion state and ignoring chunks after `end_of_stream`, but the duplicate delivery appears to violate the ext_proc contract.

## Additional note

A second failure mode was observed under the same conditions (`unexpected end of JSON input`, suggesting the body was parsed before all chunks arrived), which may be a separate delivery anomaly. Kept out of scope here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.