[Usage]: Benchmark Qwen3-FP8 series models on H800 but got unexpected low performance

System Info

System Information:

• OS: Linux
• Python version: 3.10
• CUDA version: 12.8
• GPU model(s): H800
• Driver version: 570.133.20
• TensorRT-LLM version: nvcr.io/nvidia/tensorrt-llm/release:1.1.0rc1

Detailed output:
I followed the guide https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve/run-benchmark-with-trtllm-serve.html to benchmark the default performance on H800 with Qwen/Qwen3-30B-A3B-FP8 model.

Here is the steps I performed:

docker run --runtime nvidia --gpus all --ipc=host --name tensorrt-llm --rm  -v /mnt/huggingface_hub/:/root/.cache/huggingface_hub  --env "HF_HOME=/root/.cache/huggingface_hub"  --env "HF_HUB_OFFLINE=1"  -p 8000:8000  -it nvcr.io/nvidia/tensorrt-llm/release:1.1.0rc1 /bin/bash

In the container, I edit the config as following:

enable_attention_dp: false
cuda_graph_config:
  enable_padding: true
  max_batch_size: 1024

And launched the server by:

trtllm-serve serve Qwen/Qwen3-30B-A3B-FP8  --host 0.0.0.0  --port 8000  --max_batch_size 1024  --tp_size 1     --extra_llm_api_options /tmp/config.yml

I used the vLLM benchmark_serving scripts to benchmark the server:

python vllm/benchmarks/benchmark_serving.py --model Qwen/Qwen3-30B-A3B-FP8 --base-url http://0.0.0.0:8000 --dataset-name sharegpt --dataset-path /nvme1/shared/ShareGPT_V3_unfiltered_cleaned_split.json --ignore-eos --num-prompts 10000

The results showed as below, which is much lower than I expected. Did I do something wrong?

Namespace(backend='vllm', base_url='http://0.0.0.0:8000', host='127.0.0.1', port=8000, endpoint='/v1/completions', dataset_name='sharegpt', dataset_path='/nvme1/shared/ShareGPT_V3_unfiltered_cleaned_split.json', no_stream=False, max_concurrency=None, model='Qwen/Qwen3-30B-A3B-FP8', tokenizer=None, use_beam_search=False, num_prompts=10000, logprobs=None, request_rate=inf, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, save_detailed=False, append_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=True, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, request_id_prefix='benchmark-serving', custom_output_len=256, custom_skip_chat_template=False, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, random_input_len=1024, random_output_len=128, random_range_ratio=0.0, random_prefix_len=0, hf_subset=None, hf_split=None, hf_output_len=None, top_p=None, top_k=None, min_p=None, temperature=None, tokenizer_mode='auto', served_model_name=None, lora_modules=None, ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None)
Starting initial single prompt test run...
Initial test run completed. Starting main benchmark run...
Traffic request rate: inf RPS.
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|████████████████████████████████████████████████████████████████████████████████| 10000/10000 [12:44<00:00, 13.08it/s]
============ Serving Benchmark Result ============
Successful requests:                     10000     
Benchmark duration (s):                  764.43    
Total input tokens:                      2205227   
Total generated tokens:                  2028045   
Request throughput (req/s):              13.08     
Output token throughput (tok/s):         2653.01   
Total Token throughput (tok/s):          5537.80   
---------------Time to First Token----------------
Mean TTFT (ms):                          411352.61 
Median TTFT (ms):                        456690.74 
P99 TTFT (ms):                           735033.44 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          294.92    
Median TPOT (ms):                        301.68    
P99 TPOT (ms):                           537.27    
---------------Inter-token Latency----------------
Mean ITL (ms):                           292.94    
Median ITL (ms):                         287.58    
P99 ITL (ms):                            1119.66   
==================================================

How would you like to use TensorRT-LLM

I want to run inference of a [specific model](put Hugging Face link here). I don't know how to integrate it with TensorRT-LLM or optimize it for my use case.

Specific questions:

• Model:
• Use case (e.g., chatbot, batch inference, real-time serving):
• Expected throughput/latency requirements:
• Multi-GPU setup needed:

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start with the documented trtllm-serve serve command and the /tmp/config.yml options, then inspect vllm/benchmarks/benchmark_serving.py and reproduce the reported Qwen/Qwen3-30B-A3B-FP8 run. Compare the resulting throughput and latency with the expected H800 benchmark; the issue is complete when the cause of the low performance or a confirmed configuration problem is documented.