[AutoDeploy]: Investigate streaming runtime knobs and apply where applicable
Open
@tcherckez-nvidia is already working on this.
Since Jun 14, 2026.
AutoDeploy
feature request
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
🚀 The feature, motivation and pitch
Follow up on https://github.com/NVIDIA/TensorRT-LLM/pull/14943
Investigate meaning and impact of
self.llm_args.stream_interval
self.llm_args.batch_wait_timeout_ms
self.llm_args.batch_wait_timeout_iters
self.llm_args.batch_wait_max_tokens_ratio
When should we set each of these to non-default values?
FYI - in examples/configs/curated/ only stream_interval is ever set. The others might be less useful, but we should still try to understand their purpose.
stream_interval was already discovered to be beneficial for gpt-oss and llama3.1.
Odds are it will also be beneficial for nanov3 - this is a good place to start with (quick win).
Done criteria:
- Understand meaning of each of the above knobs
- Measure impact and tune to optimal values
- Issue PR to update configs for all applicable BTK models (update model registry configs)
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.