[Bug]: function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED

System Info

CPU: Intel(R) Xeon(R) 6960P
GPU: B200 * 8
OS: Ubuntu 22.04
PyTorch: 2.8.0a0+34c6371d24.nv25.8
TRTLLM: 1.2.0rc1
CUDA: 13.0

Who can help?

Hi, @laikhtewari @kaiyux

When I was running the TRTLLM with the example benchmark script for profiling, I found there is an error from the CUPTI, which blocks us to get the torch profiler json file. Have you known how to fix the issue ?
Here is command of my running. Main part of the below command is coming from the TRTLLM official doc: https://github.com/NVIDIA/TensorRT-LLM/blob/7b82ba90da10ea1d0127d37228bda094774ce550/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md#b200-max-throughput-for-r1-0528-with-fp8-kv-cache

cat >./extra-llm-api-config.yml <<EOF
cuda_graph_config:
  enable_padding: true
  batch_sizes:
  - 896
  - 512
  - 256
  - 128
  - 64
  - 32
  - 16
  - 8
  - 4
  - 2
  - 1
print_iter_log: false
enable_attention_dp: true
kv_cache_config:
  dtype: fp8
EOF

model=nvidia/DeepSeek-R1-FP4
model_path=/data/models/DeepSeek-R1-FP4
YOUR_DATA_PATH=./dataset.txt

TLLM_PROFILE_START_STOP=300-500 nsys profile \
  -o trace -f true \
  -t 'cuda,nvtx,python-gil' -c cudaProfilerApi \
  --cuda-graph-trace node \
  -e TLLM_PROFILE_RECORD_GC=1,TLLM_LLMAPI_ENABLE_NVTX=1,TLLM_TORCH_PROFILE_TRACE=trace.json \
  --trace-fork-before-exec=true \
trtllm-bench --model ${model}                     \
     --model_path ${model_path}                   \
     throughput                                   \
     --dataset ${YOUR_DATA_PATH}                  \
     --tp 8  --ep 8                               \
     --extra_llm_api_options ./extra-llm-api-config.yml \
     --max_batch_size 896                               \
     --max_num_tokens 2048                              \
     --kv_cache_free_gpu_mem_fraction 0.93              \
     --concurrency 7168                                 \
     --num_requests 14336                               \
     2>&1 | tee log.high.throughput.log

Here is the log I got:

This failure blocks us to get the torch profiler json file, so that we cannot get the op shape information.
Could you help take a look?

Thank you.
CC: @PerryZhang01 @wuhuikx

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. running docker:

image=nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1
name=B200_trtllm

echo "launch the docker image: ${image}"
echo "launch the docker name : ${name}"

docker run -itd --rm \
  --gpus all \
  --ulimit core=0:0 \
  --ulimit memlock=-1:-1 \
  --shm-size 8G \
  --cap-add=CAP_SYS_PTRACE \
  --net host \
  -v /mnt/raid0:/mnt/raid0 \
  -v /home:/home \
  -v /data:/data \
  -v /docker:/docker \
  --name "${name}" \
  --cap-add=SYS_NICE \
  --entrypoint /bin/bash \
  "${image}"

echo "docker attach ${name}"
docker attach "${name}"

2. running command:

cat >./extra-llm-api-config.yml <<EOF
cuda_graph_config:
  enable_padding: true
  batch_sizes:
  - 896
  - 512
  - 256
  - 128
  - 64
  - 32
  - 16
  - 8
  - 4
  - 2
  - 1
print_iter_log: false
enable_attention_dp: true
kv_cache_config:
  dtype: fp8
EOF

model=nvidia/DeepSeek-R1-FP4
model_path=/data/models/DeepSeek-R1-FP4
YOUR_DATA_PATH=./dataset.txt

TLLM_PROFILE_START_STOP=300-500 nsys profile \
  -o trace -f true \
  -t 'cuda,nvtx,python-gil' -c cudaProfilerApi \
  --cuda-graph-trace node \
  -e TLLM_PROFILE_RECORD_GC=1,TLLM_LLMAPI_ENABLE_NVTX=1,TLLM_TORCH_PROFILE_TRACE=trace.json \
  --trace-fork-before-exec=true \
trtllm-bench --model ${model}                     \
     --model_path ${model_path}                   \
     throughput                                   \
     --dataset ${YOUR_DATA_PATH}                  \
     --tp 8  --ep 8                               \
     --extra_llm_api_options ./extra-llm-api-config.yml \
     --max_batch_size 896                               \
     --max_num_tokens 2048                              \
     --kv_cache_free_gpu_mem_fraction 0.93              \
     --concurrency 7168                                 \
     --num_requests 14336                               \
     2>&1 | tee log.high.throughput.log

Expected behavior

Get the torch profiler json file

actual behavior

[11/11/2025-09:12:55] [TRT-LLM] [RANK 0] [I] Setting global_steady_clock_offset: 0.0 seconds for rank 0
[11/11/2025-09:12:55] [TRT-LLM] [RANK 0] [I] Setting PyTorch memory fraction to 0.4825851851395587 (86.0748291015625 GiB)
[11/11/2025-09:12:55] [TRT-LLM] [I] get signal from executor worker
[11/11/2025-09:12:55] [TRT-LLM] [I] Setting up for warmup...
[11/11/2025-09:12:55] [TRT-LLM] [I] Running warmup.
[11/11/2025-09:12:55] [TRT-LLM] [I] Starting benchmarking async task.
[11/11/2025-09:12:55] [TRT-LLM] [I] Starting benchmark...
^MBenchmarking: 0%| | 0/2 [00:00<?, ?it/s][11/11/2025-09:12:55] [TRT-LLM] [I] Request submission complete. [count=2, time=0.0000s, rate=121285.63 req/s]
WARNING:2025-11-11 09:13:04 6353:9719 init.cpp:107] function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED (39)
WARNING:2025-11-11 09:13:04 6353:9719 init.cpp:108] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2025-11-11 09:13:04 6353:9719 init.cpp:110] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti
ERROR: External init callback must run in same thread as registerClient (-2084583744 != 1847302016)
WARNING:2025-11-11 09:13:04 6337:9717 init.cpp:107] function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED (39)
WARNING:2025-11-11 09:13:04 6337:9717 init.cpp:108] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2025-11-11 09:13:04 6337:9717 init.cpp:110] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti
ERROR: External init callback must run in same thread as registerClient (1606403776 != 1231492992)

additional notes

None

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.