LLama 3.1 405B FP8 perf low on B200

System Info

• CPU architecture: x86_64
• GPU properties
  • GPU Name: B200
  • GPU memory size: 180GB per GPU
• Libraries
  • TRTLLM v0.20.0rc0
• nVidia driver version
  • OS: Ubuntu 22.04.5 LTS
  • Driver Version: 570.133.20

Who can help?

@kaiyux, I followed https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/performance/perf-overview.md#for-systems-other-than-gh200 to benchmark https://huggingface.co/nvidia/Llama-3.1-405B-Instruct-FP8 with ISL 124, OSL 2048, number of prompt 3,000 with TP 8, PP 1.

Here is what I got. You can see the output throughput is 3,396 token per second. I ran the same things on H200 and I got 6,051 token per second which is aligned with NV published data on H200.

B200 is supposed to have a better FP8 performance than H200 due to higher flops and HBM mem bw.
I wonder if there is something related with FP8 on Blackwell undocumented.
Can you let me know if there is anything I can do to get a better perf on B200 for Llama 3.1 405B FP8? Thanks.

===========================================================

ENGINE DETAILS |   |   |   |  

Model:                  /models/nvidia-Llama-3.1-405B-Instruct-FP8 |  
Model Path:             None |   |   |   |  
Engine Directory:       /models/nvidia-Llama-3.1-405B-Instruct-FP8/tp_8_pp_1
TensorRT-LLM Version:   0.20.0rc0 |   |   |  
Dtype:                  bfloat16 |   |   |   |  
KV Cache Dtype:         FP8 |   |   |   |  
Quantization:           FP8 |   |   |   |  
Max Input Length:       1024 |   |   |   |  
Max Sequence Length:    2176 |   |   |  
  |   |   |   |   |  

REQUEST DETAILS |   |   |   |  

Number of requests:             3000 |   |   |  
Number of concurrent requests:  2178.7575 |   |  
Average Input Length (tokens):  128.0000 |   |  
Average Output Length (tokens): 2048.0000 |   |  

WORLD + RUNTIME INFORMATION |   |   |  

TP Size:                8 |   |   |   |  
PP Size:                1 |   |   |   |  
EP Size:                None |   |   |   |  
Max Runtime Batch Size: 2048 |   |   |  
Max Runtime Tokens:     2048 |   |   |  
Scheduling Policy:      GUARANTEED_NO_EVICT |   |  
KV Memory Percentage:   90.00% |   |   |  
Issue Rate (req/sec):   3.3398E+14 |   |   |  
  |   |   |   |   |  

PERFORMANCE OVERVIEW |   |   |  

Request Throughput (req/sec):                     1.6583 |   |  
Total Output Throughput (tokens/sec):             3396.1539 |  
Per User Output Throughput (tokens/sec/user):     1.7509 |  
Per GPU Output Throughput (tokens/sec/gpu):       424.5192 |  
Total Token Throughput (tokens/sec):              3608.4135 |   |  
Total Latency (ms):                               1809105.3014 |   |  
Average request latency (ms):                     1313867.2498 |   |  
  |   |   |   |   |  
-- Request Latency Breakdown (ms) ----------------------- |  
  |   |   |   |   |  
[Latency] P50    : 954688.7268 |   |   |  
[Latency] P90    : 1786663.6796 |   |   |  
[Latency] P95    : 1796832.9017 |   |   |  
[Latency] P99    : 1806071.1491 |   |   |  
[Latency] MINIMUM: 840220.8239 |   |   |  
[Latency] MAXIMUM: 1808104.6117 |   |   |  
[Latency] AVERAGE: 1313867.2498 |   |   |  

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Steps to reproduce the behaviors:

1. python benchmarks/cpp/prepare_dataset.py --tokenizer=nvidia/Llama-3.1-405B-Instruct-FP8 --stdout token-norm-dist --num-requests=3000 --input-mean=128 --output-mean=2048 --input-stdev=0 --output-stdev=0 > dataset_128_2048
2. trtllm-bench --model nvidia/Llama-3.1-405B-Instruct-FP8 build --tp_size 8 --pp_size 1 --quantization FP8 --dataset ./dataset_128_2048
3. trtllm-bench --model nvidia/Llama-3.1-405B-Instruct-FP8 throughput --dataset $dataset_file --engine_dir $engine_dir

Expected behavior

The output throughput on B200 should be higher than 6K token/s which is H200 perf.

actual behavior

I got 3.4K token per second on B200

additional notes

N/A