NVIDIA / NVIDIA/TensorRT-Edge-LLM
Experimental OpenAI server: SSE chunks missing required `object` field
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
The experimental.server (v0.7.1) /v1/chat/completions streaming response
emits SSE chunks missing the object: "chat.completion.chunk" field that
the OpenAI spec requires for ChatCompletionChunk.
Source: experimental/server/api_server.py::_sse_chunk (lines ~290 in v0.7.1):
def _sse_chunk(response_id, delta, finish_reason=None):
choice = {"delta": delta, "index": 0}
if finish_reason:
choice["finish_reason"] = finish_reason
payload = {"id": response_id, "choices": [choice]} # missing "object"
return f"data: {json.dumps(payload)}\n\n"
The non-streaming branch (line ~188) correctly emits "object": "chat.completion",
so this is a streaming-specific omission.
Impact: Strict OpenAI-compatible clients reject every chunk. We hit this
with NVIDIA AIPerf 0.8.0 (ValueError: Unsupported OpenAI object type: None,
50/50 records lost) when benchmarking Qwen3-8B-NVFP4 on Jetson AGX Thor.
Reproduce:
curl -N http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"...","messages":[{"role":"user","content":"hi"}],"stream":true}' \
| head -3
Each data: {...} line should contain "object": "chat.completion.chunk";
in v0.7.1 it does not.
Suggested fix: one line in _sse_chunk:
payload = {
"id": response_id,
"object": "chat.completion.chunk",
"choices": [choice],
}
Environment: Jetson AGX Thor T5000, JetPack 7.1, Edge-LLM v0.7.1,
container nvcr.io/nvidia/pytorch:25.12-py3.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in experimental/server/api_server.py at _sse_chunk around line 290, then compare its payload with the non-streaming response around line 188. Run the provided curl streaming reproduction and verify that every SSE data line identifies itself as a ChatCompletionChunk and is accepted by the affected client.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100