NVIDIA / NVIDIA/TensorRT-Edge-LLM

Experimental OpenAI server: SSE chunks missing required `object` field

Open Beginner friendly
#93 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
563
Forks
135
Avg merge
14h 13m
Merged PRs (30d)
1

Description

The experimental.server (v0.7.1) /v1/chat/completions streaming response
emits SSE chunks missing the object: "chat.completion.chunk" field that
the OpenAI spec requires for ChatCompletionChunk.

Source: experimental/server/api_server.py::_sse_chunk (lines ~290 in v0.7.1):

def _sse_chunk(response_id, delta, finish_reason=None):
    choice = {"delta": delta, "index": 0}
    if finish_reason:
        choice["finish_reason"] = finish_reason
    payload = {"id": response_id, "choices": [choice]}   # missing "object"
    return f"data: {json.dumps(payload)}\n\n"

The non-streaming branch (line ~188) correctly emits "object": "chat.completion",
so this is a streaming-specific omission.

Impact: Strict OpenAI-compatible clients reject every chunk. We hit this
with NVIDIA AIPerf 0.8.0 (ValueError: Unsupported OpenAI object type: None,
50/50 records lost) when benchmarking Qwen3-8B-NVFP4 on Jetson AGX Thor.

Reproduce:

curl -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"...","messages":[{"role":"user","content":"hi"}],"stream":true}' \
  | head -3

Each data: {...} line should contain "object": "chat.completion.chunk";
in v0.7.1 it does not.

Suggested fix: one line in _sse_chunk:

payload = {
    "id": response_id,
    "object": "chat.completion.chunk",
    "choices": [choice],
}

Environment: Jetson AGX Thor T5000, JetPack 7.1, Edge-LLM v0.7.1,
container nvcr.io/nvidia/pytorch:25.12-py3.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in experimental/server/api_server.py at _sse_chunk around line 290, then compare its payload with the non-streaming response around line 188. Run the provided curl streaming reproduction and verify that every SSE data line identifies itself as a ChatCompletionChunk and is accepted by the affected client.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.