Function tooling like in vllm
Open
@JunyiXu-nv is already working on this.
Since Dec 23, 2025.
feature request
Investigating
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Hi there!
Is there a possibility to use this feature like in vllm https://docs.vllm.ai/en/stable/features/tool_calling.html
in TensorRT-LLM?
vllm serve deepseek--enable-auto-tool-choice --tool-call-parser deepseek_v3 --chat-template tool_chat_template_deepseekr1.jinja
https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_deepseekv3.jinja
I did not find it in docs.
We use nvidia/DeepSeek-R1-0528-FP4 and nvidia/DeepSeek-V3-0324-FP4
trtllm-serve
--backend pytorch \
--tp_size 8 \
--ep_size 2 \
--max_num_tokens 32768 \
--trust_remote_code \
--extra_llm_api_options /root/extra-llm-api-config.yml \
--kv_cache_free_gpu_memory_fraction 0.85'
extra-llm-api-config.yml
cuda_graph_config: {}
moe_backend: TRTLLM
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
request
google adk->litellm->trtllm-serve
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.