Best performance on DeepSeek-R1 for H200 reproduce failed

We ran benckmarks for H200 following How to get best performance on DeepSeek-R1 in TensorRT-LLM.

H200 min-latency experiment is good.
But H200 max-throughput failed to reproduce.

1. For --max_num_tokens 1127, OOM.
2. Then I use --max_num_tokens 1024, it works, but too slow. I can't wait for all 5120 requests finish.
3. Then I tried:

   trtllm-bench -m deepseek-ai/DeepSeek-R1 \
       throughput \
       --tp 8 \
       --ep 8 \
       --warmup 0 \
       --dataset /workspace1/dataset.txt \
       --backend pytorch \
       --max_batch_size 128 \
       --max_num_tokens 1024 \
       --num_requests 128 \
       --concurrency 1024 \
       --kv_cache_free_gpu_mem_fraction 0.7 \
       --extra_llm_api_options /workspace1/extra-llm-api-config-max-throughput.yml
   

   I got results:

   ===========================================================
   = PYTORCH BACKEND
   ===========================================================
   Model:                  deepseek-ai/DeepSeek-R1
   Model Path:             None
   TensorRT-LLM Version:   0.19.0.dev2025041500
   Dtype:                  bfloat16
   KV Cache Dtype:         None
   Quantization:           FP8_BLOCK_SCALES
   
   ===========================================================
   = REQUEST DETAILS 
   ===========================================================
   Number of requests:             128
   Number of concurrent requests:  68.0876
   Average Input Length (tokens):  1024.0000
   Average Output Length (tokens): 2048.0000
   ===========================================================
   = WORLD + RUNTIME INFORMATION 
   ===========================================================
   TP Size:                8
   PP Size:                1
   EP Size:                8
   Max Runtime Batch Size: 128
   Max Runtime Tokens:     1024
   Scheduling Policy:      GUARANTEED_NO_EVICT
   KV Memory Percentage:   70.00%
   Issue Rate (req/sec):   4.0302E+15
   
   ===========================================================
   = PERFORMANCE OVERVIEW 
   ===========================================================
   Request Throughput (req/sec):                     0.1444
   Total Output Throughput (tokens/sec):             295.8193
   Per User Output Throughput (tokens/sec/user):     7.7889
   Per GPU Output Throughput (tokens/sec/gpu):       36.9774
   Total Token Throughput (tokens/sec):              443.7290
   Total Latency (ms):                               886162.4667
   Average request latency (ms):                     471380.1377
   
   -- Request Latency Breakdown (ms) -----------------------
   
   [Latency] P50    : 498487.4187
   [Latency] P90    : 831621.6695
   [Latency] P95    : 885649.1428
   [Latency] P99    : 885674.2557
   [Latency] MINIMUM: 55684.6616
   [Latency] MAXIMUM: 885709.5088
   [Latency] AVERAGE: 471380.1377
   
   ===========================================================
   = DATASET DETAILS
   ===========================================================
   Dataset Path:         /workspace1/dataset0.txt
   Number of Sequences:  128
   
   -- Percentiles statistics ---------------------------------
   
           Input              Output           Seq. Length
   -----------------------------------------------------------
   MIN:  1024.0000          2048.0000          3072.0000
   MAX:  1024.0000          2048.0000          3072.0000
   AVG:  1024.0000          2048.0000          3072.0000
   P50:  1024.0000          2048.0000          3072.0000
   P90:  1024.0000          2048.0000          3072.0000
   P95:  1024.0000          2048.0000          3072.0000
   P99:  1024.0000          2048.0000          3072.0000
   ===========================================================
   

   Notice that the real batch size cannot reach 128, and Total Token Throughput is 35 times lower than the reference value.

I also try to modify cuda_graph_batch_sizes in extra-llm-api-config.yml, my extra config file:

pytorch_backend_config:
    use_cuda_graph: true
    cuda_graph_batch_sizes: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128]
    enable_overlap_scheduler: true
enable_attention_dp: true

And the dataset.txt is the same as described in document: ISL = 1K, OSL = 2K.

sudo nvidia-smi -pm 0; sudo nvidia-smi -pm 1; sudo nvidia-smi boost-slider --vboost 4 already ran.

export TRTLLM_DG_JIT_USE_NVCC=1 used.

My code base is 863d023f

Any suggestion?