NVIDIA / NVIDIA/TensorRT-LLM

[AutoDeploy]: Investigate streaming runtime knobs and apply where applicable

Open
#15,161 4 comments 0 reactions 1 assignee View on GitHub

@tcherckez-nvidia is already working on this.

Since Jun 14, 2026.

AutoDeploy feature request
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

🚀 The feature, motivation and pitch

Follow up on https://github.com/NVIDIA/TensorRT-LLM/pull/14943
Investigate meaning and impact of

        self.llm_args.stream_interval
         self.llm_args.batch_wait_timeout_ms
        self.llm_args.batch_wait_timeout_iters
        self.llm_args.batch_wait_max_tokens_ratio

When should we set each of these to non-default values?
FYI - in examples/configs/curated/ only stream_interval is ever set. The others might be less useful, but we should still try to understand their purpose.
stream_interval was already discovered to be beneficial for gpt-oss and llama3.1.
Odds are it will also be beneficial for nanov3 - this is a good place to start with (quick win).

Done criteria:

  • Understand meaning of each of the above knobs
  • Measure impact and tune to optimal values
  • Issue PR to update configs for all applicable BTK models (update model registry configs)
Alternatives

No response

Additional context

No response

Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.