[Bug]: Gen ai perf tool generating incorrect number of output tokens

System Info

GPU: nvidia A100
GPU memory: 40 GB

Libraries:
TensorRT-LLM branch: r25.06

nvidia driver version: 550.90.07
container: nvcr.io/nvidia/tritonserver:25.06-trtllm-python-py3

Who can help?

@kaiyux

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. start the trt llm container and load any LLM model
2. start the triton SDK container: docker run --gpus all --rm -it --net host -v /home/jaya_kommuru/:/home/jaya_kommuru/ nvcr.io/nvidia/tritonserver:25.09-py3-sdk
3. Run the genai-perf benchmark

genai-perf profile  \
 -m ensemble  \
 --backend tensorrtllm   \
 --url localhost:8001   \
 --streaming   \
 --tokenizer meta-llama/Llama-3.1-8B \
 --synthetic-input-tokens-mean 1900 \
  --synthetic-input-tokens-stddev 0 \
  --output-tokens-mean 200 \
  --output-tokens-stddev 0 \
  --streaming \
  --concurrency 5 \
  --warmup-request-count 10 \
  --measurement-interval 60000

Expected behavior

The Output Sequence Length (tokens) should have been 200 as avg, min and max as well

actual behavior

But, as shown below for Output Sequence Length (tokens) , min is 18 and avg is 199 and not exactly 200. Whats the reason for this ? ideally it should have been 200 right ?

                                     NVIDIA GenAI-Perf | LLM Metrics                                      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃                            Statistic ┃      avg ┃      min ┃      max ┃      p99 ┃      p90 ┃      p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│             Time To First Token (ms) │    98.69 │    18.15 │   697.71 │   696.17 │   569.73 │    28.76 │
│            Time To Second Token (ms) │    24.54 │     7.13 │   554.03 │   422.18 │    11.92 │     9.60 │
│                 Request Latency (ms) │ 1,985.18 │   865.14 │ 2,683.74 │ 2,561.43 │ 2,555.98 │ 1,880.42 │
│             Inter Token Latency (ms) │     9.51 │     9.27 │    12.72 │    12.06 │     9.98 │     9.32 │
│     Output Token Throughput Per User │   105.56 │    78.64 │   107.83 │   107.79 │   107.72 │   107.66 │
│                    (tokens/sec/user) │          │          │          │          │          │          │
│      Output Sequence Length (tokens) │   199.38 │    18.00 │   200.00 │   200.00 │   200.00 │   200.00 │
│       Input Sequence Length (tokens) │ 1,900.00 │ 1,900.00 │ 1,900.00 │ 1,900.00 │ 1,900.00 │ 1,900.00 │
│ Output Token Throughput (tokens/sec) │   499.41 │      N/A │      N/A │      N/A │      N/A │      N/A │
│         Request Throughput (per sec) │     2.50 │      N/A │      N/A │      N/A │      N/A │      N/A │
│                Request Count (count) │   542.00 │      N/A │      N/A │      N/A │      N/A │      N/A │
└──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

additional notes

In some cases I am seeing bigger difference in the output tokens as shown below. Ideally it should have been 200

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃                            Statistic ┃      avg ┃      min ┃      max ┃      p99 ┃      p90 ┃      p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│             Time To First Token (ms) │    47.03 │    33.11 │    58.16 │    50.20 │    49.29 │    47.56 │
│            Time To Second Token (ms) │    22.24 │    20.67 │    25.84 │    25.76 │    23.62 │    22.03 │
│                 Request Latency (ms) │ 1,826.02 │    57.61 │ 4,463.86 │ 4,461.94 │ 4,453.98 │ 4,448.07 │
│             Inter Token Latency (ms) │    32.56 │    22.08 │    44.25 │    44.10 │    43.93 │    43.78 │
│     Output Token Throughput Per User │    33.21 │    22.60 │    45.29 │    45.28 │    45.24 │    45.18 │
│                    (tokens/sec/user) │          │          │          │          │          │          │
│      Output Sequence Length (tokens) │    68.86 │     1.00 │   200.00 │   200.00 │   200.00 │   139.00 │
│       Input Sequence Length (tokens) │ 1,946.80 │ 1,939.00 │ 1,963.00 │ 1,963.00 │ 1,963.00 │ 1,945.00 │
│ Output Token Throughput (tokens/sec) │    75.35 │      N/A │      N/A │      N/A │      N/A │      N/A │
│         Request Throughput (per sec) │     1.09 │      N/A │      N/A │      N/A │      N/A │      N/A │
│                Request Count (count) │   235.00 │      N/A │      N/A │      N/A │      N/A │      N/A │
└──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start by reproducing the genai-perf profile command against the Triton TensorRT-LLM ensemble and inspect how the benchmark calculates Output Sequence Length. Compare the reported token counts with the requested --output-tokens-mean 200 and the streaming responses. Done means the metric behavior is explained or corrected so the reported values match the benchmark's intended output-token semantics.