ext_proc FULL_DUPLEX_STREAMED: duplicate RequestBody chunk with EndOfStream=true on large bodies (1.35+)
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 428
Description
## Summary
With `request_body_mode: FULL_DUPLEX_STREAMED`, Envoy can re-deliver the final `RequestBody` chunk (with `end_of_stream=true`) after the ext_proc server has already received and processed an `end_of_stream=true` chunk for the same request. This causes the ext_proc server to re-process an already-complete body, leading to parse failures.
## Versions affected
| Istio version | Envoy version | Duplicate EoS observed |
|---|---|---|
| 1.26.2 | 1.34 | No (3/3 pass) |
| 1.27.9 | 1.35.13 | Yes (~2/3 fail) |
| 1.28.8 | 1.36.9 | Yes (~4/5 fail) |
| 1.29.5 | 1.37.5 | Yes (~7/8 fail) |
The failure rate increases with newer Envoy versions, suggesting a regression introduced in 1.35.
## Reproduction
**Setup:** Single ext_proc filter with `request_body_mode: FULL_DUPLEX_STREAMED`, processing JSON inference requests.
**Trigger:** Send a large (~1MB+) request body. Multi-turn chat payloads consistently reproduce this.
```bash
# Generate a ~1.2MB JSON body
python3 -c "
import json
msgs = [{'role':'user','content':'analyze this: ' + 'x'*500}]
for i in range(20):
msgs.append({'role':'assistant','content':[{'type':'output_text','text':'y'*2000}]})
msgs.append({'role':'user','content':'check function ' + str(i)})
json.dump({'model':'gpt-5.5','input':msgs}, open('body.json','w'))
"
curl -d @body.json \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
https:///v1/responses
```
**Expected:** One `RequestBody` message with `end_of_stream=true`, followed by no further `RequestBody` messages for this request.
**Actual:** Two `RequestBody` messages with `end_of_stream=true` for the same stream. Server logs:
```
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":false} x 7
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":true} <- first EoS, processed
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":true} <- duplicate EoS
```
**Same payload without ext_proc:** 100% success (50/50 at 2.5MB), confirming the issue is in ext_proc delivery, not upstream.
## Impact
Any ext_proc server that accumulates body chunks and processes on `end_of_stream=true` will re-process the already-complete buffer with the duplicate chunk appended, causing parse failures. This affects real workloads -- multi-turn LLM conversations routinely exceed 1MB.
We have worked around this in [llm-d-inference-payload-processor#225](https://github.com/llm-d/llm-d-inference-payload-processor/pull/225) by tracking body completion state and ignoring chunks after `end_of_stream`, but the duplicate delivery appears to violate the ext_proc contract.
## Additional note
A second failure mode was observed under the same conditions (`unexpected end of JSON input`, suggesting the body was parsed before all chunks arrived), which may be a separate delivery anomaly. Kept out of scope here.
Contributor guide
Assessment
This issue has not been assessed yet.