[Bug]: [AutoDeploy] The CLI-specified max_batch_size only takes effect if max_num_tokens is also set in the CLI

System Info

The CLI-specified max_batch_size only takes effect if max_num_tokens is also set in the CLI

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

python3 benchmarks/cpp/prepare_dataset.py --stdout --tokenizer nvidia/Llama-3.1-8B-Instruct-FP8 token-norm-dist --input-mean 1000 --output-mean 2000 --input-stdev 0 --output-stdev 0 --num-requests 64 > llama_8b_1k_2k_64.inp

trtllm-bench --model nvidia/Llama-3.1-8B-Instruct-FP8 throughput --dataset /home/scratch.egeva_coreai/TensorRT-LLM/llama_8b_1k_2k.inp --backend _autodeploy --extra_llm_api_options /home/scratch.egeva_coreai/TensorRT-LLM/examples/auto_deploy/llama_8b.yaml

use this yaml:

attn_backend: flashinfer
compile_backend: torch-cudagraph
enable_chunked_prefill: true
free_mem_ratio: 0.88
max_batch_size: 1
max_seq_len: 65536
model_factory: AutoModelForCausalLM
runtime: trtllm
skip_loading_weights: false

Expected behavior

the CLI max_batch_size should win the one defined in the yaml

actual behavior

na

additional notes

na

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.