NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: standalone Responses API input, streaming, and tool continuation failures with Agentic API

Open
#19,223 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

LLM API
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

hello, i'm Francisco, maintainer of vLLM Agentic API. i'm testing streaming, conversation continuation, and tool calls with Qwen3-8B and GPT-OSS-20B against TensorRT-LLM's standalone /v1/responses. our Dynamo → TensorRT-LLM integration worked, but the standalone path exposed these issues:

  • input conversion: typed messages with string content become empty on the Qwen path; empty strings raise an error.
  • output replay: returned logprobs:null can fail input validation. summary-only, empty, or multipart reasoning items can also fail or lose content.
  • streaming: item IDs, call IDs, and output indices can disagree with the terminal response. Harmony tool-call events are missing, and incomplete responses get a response.completed event.
  • continuation: streamed responses aren't saved by the frontend path even with storage enabled. previous_response_id with store:false also drops prior history.
  • tool results: stored chat history can lose tool calls; GPT-OSS/Harmony misses some call channels/recipients and fails to pair replayed calls with their results.
  • instructions: Harmony continuation reuses old instructions and tool definitions instead of the current request's.

i have local fixes for these. the tests pass and through it works using vLLM Agentic API on an NVIDIA GB10, using TensorRT-LLM 1.3.0rc26 with patched serving code, OpenAI SDK 3.8.0, and Pydantic 2.13.4. this tested one frontend with frontend postprocessing, not a full rebuild of main.

the first draft, #19222, covers input conversion/replay and includes regression tests: 86 unit tests and pre-commit checks passed. the live results above cover the combined local fixes, not that PR alone. i'll keep streaming/storage and tool-history/Harmony fixes in separate PRs.

does this split make sense, and are any of these behaviors intentional restrictions of the standalone API?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing draft #19222 and its regression tests for input conversion and replay, then compare standalone /v1/responses behavior with the listed streaming, storage, tool-history, and Harmony cases. The remaining work is planned as separate pull requests; done means each behavior has agreed scope and corresponding regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, api, backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.