NVIDIA / NVIDIA/TensorRT-LLM

CUDA error CUBLAS_STATUS_EXECUTION_FAILED when launching Qwen2.5-VL-72B using quickstart_multimodal.py

Open
#4,910 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug Model optimization Multimodal Scale-out triaged
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
Environment
  • cuda 12.4
  • torch 2.7.0
  • tensorrt_llm 0.20.0rc3
  • GPU L20 x 4
  • Qwen2.5-VL-72B (7B has the same error)
Who can help?

No response

Information
  • The official example scripts
  • My own modified scripts
Tasks
  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)
Reproduction
Launching script

Refer to the following README.

https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/pytorch

python3 quickstart_multimodal.py --model_dir /workspace/qwen_src/Qwen2.5-VL-72B-Instruct  --modality image --prompt "Tell me what you see briefly." --media "/workspace/qwen_src/test_data/17_inp.png" --max_tokens 128 --tp_size 4
Expected behavior

We expected it to launch normally and generate text.

actual behavior
Error log
[1749031978.045350] [njxg-feed-1713759761:58398:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
/usr/local/lib/python3.10/dist-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
2025-06-04 10:13:06,579 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.20.0rc3
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 38/38 [00:00<00:00, 162.36it/s]
rank 0 using MpiPoolSession to spawn MPI processes
[1749031989.007169] [njxg-feed-1713759761:59942:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1749031989.007205] [njxg-feed-1713759761:59941:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1749031989.007206] [njxg-feed-1713759761:59943:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1749031989.007291] [njxg-feed-1713759761:59940:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
/usr/local/lib/python3.10/dist-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
2025-06-04 10:13:17,605 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-06-04 10:13:17,606 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-06-04 10:13:17,609 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-06-04 10:13:17,669 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.20.0rc3
[TensorRT-LLM] TensorRT-LLM version: 0.20.0rc3
[TensorRT-LLM] TensorRT-LLM version: 0.20.0rc3
[TensorRT-LLM] TensorRT-LLM version: 0.20.0rc3
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] Refreshed the MPI local session
Loading weights: 100%|██████████| 1529/1529 [00:18<00:00, 84.30it/s]
Model init total -- 24.51s
Loading weights: 100%|██████████| 1529/1529 [00:19<00:00, 78.58it/s]
Model init total -- 25.75s
Loading weights: 100%|██████████| 1529/1529 [00:19<00:00, 78.22it/s]
Model init total -- 25.90s
Loading weights: 100%|██████████| 1529/1529 [00:20<00:00, 75.31it/s]
Model init total -- 26.60s
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1892 [window size=60544]
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1892 [window size=60544]
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1892 [window size=60544]
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1892 [window size=60544]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 4.62 GiB for max tokens in paged KV cache (60544).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 4.62 GiB for max tokens in paged KV cache (60544).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 4.62 GiB for max tokens in paged KV cache (60544).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 4.62 GiB for max tokens in paged KV cache (60544).
[TensorRT-LLM][INFO] Detecting local TP group for rank 1
[TensorRT-LLM][INFO] Detecting local TP group for rank 2
[TensorRT-LLM][INFO] Detecting local TP group for rank 0
[TensorRT-LLM][INFO] Detecting local TP group for rank 3
[TensorRT-LLM][INFO] TP group is intra-node for rank 2
[TensorRT-LLM][INFO] TP group is intra-node for rank 1
[TensorRT-LLM][INFO] TP group is intra-node for rank 0
[TensorRT-LLM][INFO] TP group is intra-node for rank 3
2025-06-04 10:13:46,785 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-06-04 10:13:46,785 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-06-04 10:13:46,786 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-06-04 10:13:46,786 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-06-04 10:13:46,850 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-06-04 10:13:46,950 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-06-04 10:13:47,050 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-06-04 10:13:47,165 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-06-04 10:13:49,132 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
2025-06-04 10:13:49,133 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
2025-06-04 10:13:49,189 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
2025-06-04 10:13:49,195 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
2025-06-04 10:13:49,274 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
2025-06-04 10:13:49,297 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
2025-06-04 10:13:49,385 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
2025-06-04 10:13:49,452 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1614 [window size=51648]
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1614 [window size=51648]
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1614 [window size=51648]
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 1614 [window size=51648]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 3.94 GiB for max tokens in paged KV cache (51648).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 3.94 GiB for max tokens in paged KV cache (51648).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 3.94 GiB for max tokens in paged KV cache (51648).
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 3.94 GiB for max tokens in paged KV cache (51648).
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Processed requests:   0%|                                                                            | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1607, in _forward_step
    outputs = forward(scheduled_requests, self.resource_manager,
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1602, in forward
    return self.model_engine.forward(scheduled_requests,
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/utils.py", line 66, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1882, in forward
    outputs = self._forward_step(inputs, gather_ids)
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1927, in _forward_step
    logits = self.model_forward(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1915, in model_forward
    return self.model.forward(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen2vl.py", line 436, in forward
    output_prob = self.llm.forward(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 195, in forward
    output = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 162, in forward
    hidden_states, residual = decoder_layer(position_ids=position_ids,
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 101, in forward
    hidden_states = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/attention.py", line 230, in forward
    qkv = self.qkv_proj(hidden_states)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 421, in forward
    output = self.apply_linear(input, self.weight, self.bias,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 370, in apply_linear
    output = F.linear(input, self.weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 2560 n 1299 k 8192 mat1_ld 8192 mat2_ld 8192 result_ld 2560 abcType 14 computeType 68 scaleType 0
[06/04/2025-10:14:07] [TRT-LLM] [E] Encountered an error in forward function: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 2560 n 1299 k 8192 mat1_ld 8192 mat2_ld 8192 result_ld 2560 abcType 14 computeType 68 scaleType 0
Exception in thread Thread-4 (_executor_loop_overlap):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1015, in _executor_loop_overlap
    self._update_request_states(scheduled_batch)
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1646, in _update_request_states
    self._update_request_states_tp(scheduled_requests)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1620, in _update_request_states_tp
    request.move_to_next_context_chunk()
RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: Chunking is only possible during the context phase. (/home/jenkins/agent/workspace/LLM/main/L0_Test-x86_64/tensorrt_llm/cpp/include/tensorrt_llm/batch_manager/llmRequest.h:1556)
1       0x7fbdf713c730 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0x3b730) [0x7fbdf713c730]
2       0x7fbdf71d834e /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xd734e) [0x7fbdf71d834e]
3       0x7fbdf71e654e /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xe554e) [0x7fbdf71e654e]
4       0x7fbdf71ac660 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xab660) [0x7fbdf71ac660]
5       0x55b8475cdb32 /usr/bin/python3(+0x18ab32) [0x55b8475cdb32]
6       0x55b8475c439b _PyObject_MakeTpCall + 603
7       0x55b8475db7ab /usr/bin/python3(+0x1987ab) [0x55b8475db7ab]
8       0x55b8475bd702 _PyEval_EvalFrameDefault + 22066
9       0x55b8475ce38c _PyFunction_Vectorcall + 124
10      0x55b8475b897f _PyEval_EvalFrameDefault + 2223
11      0x55b8475ce38c _PyFunction_Vectorcall + 124
12      0x55b8475bac30 _PyEval_EvalFrameDefault + 11104
13      0x55b8475ce38c _PyFunction_Vectorcall + 124
14      0x55b8475b897f _PyEval_EvalFrameDefault + 2223
15      0x55b8475db731 /usr/bin/python3(+0x198731) [0x55b8475db731]
16      0x55b8475bac30 _PyEval_EvalFrameDefault + 11104
17      0x55b8475ce38c _PyFunction_Vectorcall + 124
18      0x55b8475b897f _PyEval_EvalFrameDefault + 2223
19      0x55b8475ce38c _PyFunction_Vectorcall + 124
20      0x55b8475b897f _PyEval_EvalFrameDefault + 2223
21      0x55b8475db731 /usr/bin/python3(+0x198731) [0x55b8475db731]
22      0x55b8476f07ca /usr/bin/python3(+0x2ad7ca) [0x55b8476f07ca]
23      0x55b8476e6138 /usr/bin/python3(+0x2a3138) [0x55b8476e6138]
24      0x7fc0cd118ac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fc0cd118ac3]
25      0x7fc0cd1aa850 /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7fc0cd1aa850]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1607, in _forward_step
    outputs = forward(scheduled_requests, self.resource_manager,
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1602, in forward
    return self.model_engine.forward(scheduled_requests,
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/utils.py", line 66, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1882, in forward
    outputs = self._forward_step(inputs, gather_ids)
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1927, in _forward_step
    logits = self.model_forward(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1915, in model_forward
    return self.model.forward(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen2vl.py", line 436, in forward
    output_prob = self.llm.forward(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 195, in forward
    output = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 162, in forward
    hidden_states, residual = decoder_layer(position_ids=position_ids,
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 101, in forward
    hidden_states = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/attention.py", line 230, in forward
    qkv = self.qkv_proj(hidden_states)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 421, in forward
    output = self.apply_linear(input, self.weight, self.bias,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 370, in apply_linear
    output = F.linear(input, self.weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 2560 n 1299 k 8192 mat1_ld 8192 mat2_ld 8192 result_ld 2560 abcType 14 computeType 68 scaleType 0
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1607, in _forward_step
    outputs = forward(scheduled_requests, self.resource_manager,
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1602, in forward
    return self.model_engine.forward(scheduled_requests,
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/utils.py", line 66, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1882, in forward
    outputs = self._forward_step(inputs, gather_ids)
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1927, in _forward_step
    logits = self.model_forward(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 1915, in model_forward
    return self.model.forward(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen2vl.py", line 436, in forward
    output_prob = self.llm.forward(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 195, in forward
    output = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 162, in forward
    hidden_states, residual = decoder_layer(position_ids=position_ids,
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/models/modeling_qwen.py", line 101, in forward
    hidden_states = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/attention.py", line 230, in forward
    qkv = self.qkv_proj(hidden_states)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 421, in forward
    output = self.apply_linear(input, self.weight, self.bias,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/modules/linear.py", line 370, in apply_linear
    output = F.linear(input, self.weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 2560 n 1299 k 8192 mat1_ld 8192 mat2_ld 8192 result_ld 2560 abcType 14 computeType 68 scaleType 0
[06/04/2025-10:14:07] [TRT-LLM] [E] Encountered an error in forward function: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 2560 n 1299 k 8192 mat1_ld 8192 mat2_ld 8192 result_ld 2560 abcType 14 computeType 68 scaleType 0
[06/04/2025-10:14:07] [TRT-LLM] [E] Encountered an error in forward function: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 2560 n 1299 k 8192 mat1_ld 8192 mat2_ld 8192 result_ld 2560 abcType 14 computeType 68 scaleType 0
Exception in thread Thread-4 (_executor_loop_overlap):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Exception in thread Thread-4 (_executor_loop_overlap):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1015, in _executor_loop_overlap
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._update_request_states(scheduled_batch)
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1015, in _executor_loop_overlap
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1646, in _update_request_states
    self._update_request_states(scheduled_batch)
  File "/usr/local/lib/python3.10/dist-packages/nvtx/nvtx.py", line 101, in inner
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1646, in _update_request_states
    self._update_request_states_tp(scheduled_requests)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1620, in _update_request_states_tp
    self._update_request_states_tp(scheduled_requests)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1620, in _update_request_states_tp
    request.move_to_next_context_chunk()
RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: Chunking is only possible during the context phase. (/home/jenkins/agent/workspace/LLM/main/L0_Test-x86_64/tensorrt_llm/cpp/include/tensorrt_llm/batch_manager/llmRequest.h:1556)
1       0x7f25a5f3c730 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0x3b730) [0x7f25a5f3c730]
2       0x7f25a5fd834e /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xd734e) [0x7f25a5fd834e]
3       0x7f25a5fe654e /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xe554e) [0x7f25a5fe654e]
4       0x7f25a5fac660 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xab660) [0x7f25a5fac660]
5       0x55a4378cbb32 /usr/bin/python3(+0x18ab32) [0x55a4378cbb32]
6       0x55a4378c239b _PyObject_MakeTpCall + 603
7       0x55a4378d97ab /usr/bin/python3(+0x1987ab) [0x55a4378d97ab]
8       0x55a4378bb702 _PyEval_EvalFrameDefault + 22066
9       0x55a4378cc38c _PyFunction_Vectorcall + 124
10      0x55a4378b697f _PyEval_EvalFrameDefault + 2223
11      0x55a4378cc38c _PyFunction_Vectorcall + 124
12      0x55a4378b8c30 _PyEval_EvalFrameDefault + 11104
13      0x55a4378cc38c _PyFunction_Vectorcall + 124
14      0x55a4378b697f _PyEval_EvalFrameDefault + 2223
15      0x55a4378d9731 /usr/bin/python3(+0x198731) [0x55a4378d9731]
16      0x55a4378b8c30 _PyEval_EvalFrameDefault + 11104
17      0x55a4378cc38c _PyFunction_Vectorcall + 124
18      0x55a4378b697f _PyEval_EvalFrameDefault + 2223
19      0x55a4378cc38c _PyFunction_Vectorcall + 124
20      0x55a4378b697f _PyEval_EvalFrameDefault + 2223
21      0x55a4378d9731 /usr/bin/python3(+0x198731) [0x55a4378d9731]
22      0x55a4379ee7ca /usr/bin/python3(+0x2ad7ca) [0x55a4379ee7ca]
23      0x55a4379e4138 /usr/bin/python3(+0x2a3138) [0x55a4379e4138]
24      0x7f287ba9aac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f287ba9aac3]
25      0x7f287bb2c850 /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f287bb2c850]
    request.move_to_next_context_chunk()
RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: Chunking is only possible during the context phase. (/home/jenkins/agent/workspace/LLM/main/L0_Test-x86_64/tensorrt_llm/cpp/include/tensorrt_llm/batch_manager/llmRequest.h:1556)
1       0x7f72d2a3c730 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0x3b730) [0x7f72d2a3c730]
2       0x7f72d2ad834e /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xd734e) [0x7f72d2ad834e]
3       0x7f72d2ae654e /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xe554e) [0x7f72d2ae654e]
4       0x7f72d2aac660 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/bindings.cpython-310-x86_64-linux-gnu.so(+0xab660) [0x7f72d2aac660]
5       0x5559fa52db32 /usr/bin/python3(+0x18ab32) [0x5559fa52db32]
6       0x5559fa52439b _PyObject_MakeTpCall + 603
7       0x5559fa53b7ab /usr/bin/python3(+0x1987ab) [0x5559fa53b7ab]
8       0x5559fa51d702 _PyEval_EvalFrameDefault + 22066
9       0x5559fa52e38c _PyFunction_Vectorcall + 124
10      0x5559fa51897f _PyEval_EvalFrameDefault + 2223
11      0x5559fa52e38c _PyFunction_Vectorcall + 124
12      0x5559fa51ac30 _PyEval_EvalFrameDefault + 11104
13      0x5559fa52e38c _PyFunction_Vectorcall + 124
14      0x5559fa51897f _PyEval_EvalFrameDefault + 2223
15      0x5559fa53b731 /usr/bin/python3(+0x198731) [0x5559fa53b731]
16      0x5559fa51ac30 _PyEval_EvalFrameDefault + 11104
17      0x5559fa52e38c _PyFunction_Vectorcall + 124
18      0x5559fa51897f _PyEval_EvalFrameDefault + 2223
19      0x5559fa52e38c _PyFunction_Vectorcall + 124
20      0x5559fa51897f _PyEval_EvalFrameDefault + 2223
21      0x5559fa53b731 /usr/bin/python3(+0x198731) [0x5559fa53b731]
22      0x5559fa6507ca /usr/bin/python3(+0x2ad7ca) [0x5559fa6507ca]
23      0x5559fa646138 /usr/bin/python3(+0x2a3138) [0x5559fa646138]
24      0x7f75a85a1ac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f75a85a1ac3]
25      0x7f75a8633850 /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f75a8633850]
Processed requests:   0%|                                                                            | 0/1 [01:15<?, ?it/s]
Traceback (most recent call last):
  File "/workspace/zzz_tmp/tllmtorch_tllmtorch/quickstart_multimodal.py", line 119, in <module>
    main()
  File "/workspace/zzz_tmp/tllmtorch_tllmtorch/quickstart_multimodal.py", line 108, in main
    outputs = llm.generate(inputs, sampling_params)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/llmapi/llm.py", line 253, in generate
    future.result()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/executor/result.py", line 477, in result
    self._result_step(timeout)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/executor/result.py", line 458, in _result_step
    response = self.queue.get(timeout=timeout)
  File "/usr/lib/python3.10/queue.py", line 171, in get
    self.not_empty.wait()
  File "/usr/lib/python3.10/threading.py", line 320, in wait
    waiter.acquire()
KeyboardInterrupt
^C^C^C^C^CError in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
KeyboardInterrupt

Original exception was:
Traceback (most recent call last):
  File "/workspace/zzz_tmp/tllmtorch_tllmtorch/quickstart_multimodal.py", line 119, in <module>
    main()
  File "/workspace/zzz_tmp/tllmtorch_tllmtorch/quickstart_multimodal.py", line 108, in main
    outputs = llm.generate(inputs, sampling_params)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/llmapi/llm.py", line 253, in generate
    future.result()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/executor/result.py", line 477, in result
    self._result_step(timeout)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/executor/result.py", line 458, in _result_step
    response = self.queue.get(timeout=timeout)
  File "/usr/lib/python3.10/queue.py", line 171, in get
    self.not_empty.wait()
  File "/usr/lib/python3.10/threading.py", line 320, in wait
    waiter.acquire()
KeyboardInterrupt
^C
additional notes

The same error can be found when launching both Qwen2.5-VL-7B and Qwen2.5-VL-7B.
But when setting --tp_size 1, the Qwen2.5-VL-7B behaves ok.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the quickstart_multimodal.py command from the examples/pytorch README using the listed CUDA, PyTorch, TensorRT-LLM, and L20 environment. Trace the failure through modeling_qwen2vl.py and modeling_qwen.py, then verify that the Qwen2.5-VL example generates text on four GPUs without the CUBLAS execution error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.