The performance of pytorch backend is worse than that of C backend

System Info

CPU x86_64

GPU NVIDIA H20

TensorRT branch: v0.20.0

NVIDIA-SMI 535.216.01 Driver Version: 535.216.01 CUDA Version: 12.2

Who can help?

@Tracin @kaiyux

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I just use pytorch backend to run deepseek-coder-33b-instruct, but the test speed is much slower than Trt's C backend, it is so wired.
I want to know how to get a better performance.

1. For Trt C backend:
I use trt_llm_v0.16.0, and 2 GPUs for tp, and FP8 quantization.
Here are the commands:

The quantizztion command:

python3 /data/trt_llm_code/trt_llm_v0.16.0/tensorrtllm_backend/tensorrt_llm/examples/quantization/quantize.py --model_dir /data/base_models/deepseek-33b \
    --dtype bfloat16 \
    --qformat fp8 \
    --kv_cache_dtype fp8 \
    --output_dir /data/trt_llm_quantize_files/nv-hopper/trt-v16-deepseek-33b-fp8/2-gpu \
    --tp_size 2

The trtllm-build command:

trtllm-build --checkpoint_dir /data/trt_llm_quantize_files/nv-hopper/trt-v16-deepseek-33b-fp8/2-gpu \
    --output_dir /data/trt_llm_engine_files/nv-hopper/trt_v16_engines-deepseek-33b-fp8-bs64/2-gpu \
    --max_input_len 16000 \
    --paged_kv_cache enable \
    --use_fp8_context_fmha enable \
    --use_paged_context_fmha enable \
    --gpt_attention_plugin auto \
    --gemm_plugin fp8 \
    --reduce_fusion enable \
    --tokens_per_block 32 \
    --multiple_profiles enable \
    --max_batch_size 64  \
    --workers 2

2. For Trt pytorch backend:
I use the ModelOpt v0.29.0 to quantize deepseek-coder-33b-instruct for FP8, and 2 GPUs for tp.
Here are the commands:

The quantization command:

python3 hf_ptq.py --pyt_ckpt_path=//data/hanleyzhao/models_test/base_models/deepseek-33b \
--export_path=/data/trt_pytorch_quantize_files/trt-pytorch-deepseek-33b-fp8 \
--qformat=fp8 \
--kv_cache_qformat fp8 \
--trust_remote_code \
--export_fmt=hf

The trtllm-serve launch command:

trtllm-serve \
  /data/trt_pytorch_quantize_files/trt-pytorch-deepseek-33b-fp8 \
  --host 0.0.0.0 \
  --port 8000 \
  --backend pytorch \
  --max_batch_size 128 \
  --max_num_tokens 1024 \
  --max_seq_len 16000 \
  --tp_size 2 \
  --ep_size 1 \
  --pp_size 1 \
  --kv_cache_free_gpu_memory_fraction 0.9 \
  --extra_llm_api_options /data/run_pytorch_server_cmd/extra-llm-api-config.yml

The extra-llm-api-config.yml config:

cuda_graph_config:
  padding_enabled: true
  batch_sizes:
    - 1
    - 2
    - 4
    - 8
    - 16
    - 32
    - 64
    - 128
print_iter_log: true
enable_attention_dp: false

Expected behavior

I thought the performance of pytorch backend would be similar to or better than C backend.
But pytorch backend is actually much worse.

actual behavior

I made a simple request test.

1. For Trt C backend:
The request command:

time curl -X POST localhost:50000/v2/models/ensemble/generate_stream \
-d '{"text_input": "def quick_sort", "max_tokens": 100, "bad_words": "", "stop_words": "", "stream": false}'

data: {"model_name":"tensorrt_llm_bls","model_version":"1","text_output":"(arr):\n    if len(arr) <= 1:\n        return arr\n    pivot = arr[len(arr) // 2]\n    left = [x for x in arr if x < pivot]\n    middle = [x for x in arr if x == pivot]\n    right = [x for x in arr if x > pivot]\n    return quick_sort(left) + middle + quick_sort(right)\n\n"}

real    0m0.831s
user    0m0.002s
sys     0m0.004s

The model inference took 0.83s for 100 tokens.

2. For Trt pytorch backend:

time curl http://localhost:50000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
      "model": "trt-pytorch-deepseek-33b-fp8",
      "prompt": "def quick_sort",
      "max_tokens": 100,
      "temperature": 0.1,
      "stream": false
  }'

{"id":"cmpl-e2d0d748eb3f44e89b841c54d90b97ff","object":"text_completion","created":1752671555,"model":"trt-pytorch-deepseek-33b-v1.2-250703-fp8","choices":[{"index":0,"text":"(arr):\n    if len(arr) <= 1:\n        return arr\n    pivot = arr[len(arr) // 2]\n    left = [x for x in arr if x < pivot]\n    middle = [x for x in arr if x == pivot]\n    right = [x for x in arr if x > pivot]\n    return quick_sort(left) + middle + quick_sort(right)\n\n","logprobs":null,"context_logits":null,"finish_reason":"length","stop_reason":null,"disaggregated_params":null}],"usage":{"prompt_tokens":5,"total_tokens":105,"completion_tokens":100}}

real    0m4.209s
user    0m0.003s
sys     0m0.003s

The model inference took 4.2s for 100 tokens.

There is a huge performance gap between the two.

additional notes

Please help me to find out what the problem is that causes such a big performance difference.
How I can solve it.
Thank you