LLM API auto_parallel feature Error

System Info

CPU Architecture x86 (Intel(R) Xeon(R) w9-3475X)
CPU Host memory size 1T
GPU Properties
GPU Name ADA 6000 (4 GPUs in the workstation)
GPU Memory size 4 X (48 GB, for each GPU)
Libraries
TensorRT LLM version 0.20.0
CUDA version 12.8
Container version nvcr.io/nvidia/tensorrt-llm/release:0.20.0
Nvidia driver version 570.133.07
OS Ubuntu 22.04.5 LTS

Who can help?

@ncomly-nvidia

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

docker run instruction (the image that I used is nvcr.io/nvidia/tensorrt-llm/release:0.20.0):
sudo docker run --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --gpus=all --runtime=nvidia -p 8080:8080 --mount type=bind,src=/home/dto-7,dst=/usr/dto-7 -w /usr/dto-7 tensor-rt-llm:0.20.0

Tried loading this model (cached locally):
https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP8

Python Code (from auto_parallel documentatation):
from tensorrt_llm import LLM
llm = LLM(
model='/usr/dto-7/.cache/huggingface/hub/models--nvidia--Llama-3.3-70B-Instruct-FP8/snapshots/26577be1645bd323bc3b2ddc8bcace3514186af7',
# Enable auto parallelism
auto_parallel=True,
auto_parallel_world_size=4
)

Expected behavior

The code should create a model inferencing object, though it fails.

actual behavior

Error:
2025-07-25 21:05:50,070 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-07-25 21:05:50,152 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-07-25 21:05:50,152 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-07-25 21:05:50,180 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
Loading Model: [1/2] Loading HF model to memory
1526it [00:42, 35.79it/s]
1526it [00:42, 35.93it/s]
1526it [00:42, 35.89it/s]
1471it [00:42, 34.69it/s]Time: 43.253s
Loading Model: [2/2] Building TRT-LLM engine
1526it [00:43, 35.00it/s]
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 821, in _node_build_task
model_loader(engine_dir=engine_dir)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 263, in call
pipeline()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 203, in call
self.step_forward()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 232, in step_forward
self.step_handlersself.counter
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 39, in wrapper
raise e
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 821, in _node_build_task
model_loader(engine_dir=engine_dir)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 263, in call
pipeline()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 203, in call
self.step_forward()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 232, in step_forward
self.step_handlersself.counter
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 39, in wrapper
raise e
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 821, in _node_build_task
model_loader(engine_dir=engine_dir)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 263, in call
pipeline()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 203, in call
self.step_forward()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 232, in step_forward
self.step_handlersself.counter
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 39, in wrapper
raise e
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 821, in _node_build_task
model_loader(engine_dir=engine_dir)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 263, in call
pipeline()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 203, in call
self.step_forward()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 232, in step_forward
self.step_handlersself.counter
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 39, in wrapper
raise e
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/utils.py", line 35, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py", line 538, in _build_engine
self._engine = build(self.model, copied_build_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/builder.py", line 1305, in build
model(**inputs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 1085, in forward
hidden_states = self.transformer.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 381, in forward
hidden_states = self.layers.forward(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 636, in forward
hidden_states = layer(
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/llama/model.py", line 200, in forward
attention_output = self.attention(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/layers/attention.py", line 833, in forward
qkv = [gemm(hidden_states) for gemm in qkv_gemm]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/module.py", line 52, in call
output = self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/quantization/layers.py", line 1433, in forward
alpha = self.weights_scaling_factor.raw_value * self.activation_scaling_factor.raw_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 203, in raw_value
assert isinstance(
^^^^^^^^^^^
AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value

AssertionError Traceback (most recent call last)
Cell In[2], line 1
----> 1 llm = LLM(
2 model='/usr/dto-7/.cache/huggingface/hub/models--nvidia--Llama-3.3-70B-Instruct-FP8/snapshots/26577be1645bd323bc3b2ddc8bcace3514186af7',
3 # Enable auto parallelism
4 auto_parallel=True,
5 auto_parallel_world_size=4
6 )

File /usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py:176, in LLM.init(self, model, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, tensor_parallel_size, dtype, revision, tokenizer_revision, **kwargs)
174 if self.mpi_session is not None:
175 self.mpi_session.shutdown()
--> 176 raise e
178 exception_handler.register(self, 'shutdown')
179 atexit.register(LLM._shutdown_wrapper, weakref.ref(self))

File /usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py:171, in LLM.init(self, model, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, tensor_parallel_size, dtype, revision, tokenizer_revision, **kwargs)
168 self.runtime_context: Optional[_ModelRuntimeContext] = None
169 self.llm_build_stats = LlmBuildStats()
--> 171 self._build_model()
173 except Exception as e:
174 if self.mpi_session is not None:

File /usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py:517, in LLM._build_model(self)
511 def _build_model(self):
512 model_loader = CachedModelLoader(self.args,
513 mpi_session=self.mpi_session,
514 workspace=self.workspace,
515 llm_build_stats=weakref.proxy(
516 self.llm_build_stats))
--> 517 self._engine_dir, self._hf_model_dir = model_loader()
518 # update the model_dir to a local dir for the runtime, such as tokenizer loading.
519 if self._engine_dir is not None:

File /usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py:674, in CachedModelLoader.call(self)
671 self.llm_build_stats.engine_dir = self.model_loader.model_obj.model_dir
672 return self.llm_build_stats.engine_dir, self._hf_model_dir
--> 674 return self._build_model(), self._hf_model_dir

File /usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py:803, in CachedModelLoader._build_model(self)
800 self.llm_build_stats.cache_info = "The cache root directory is too small."
802 if not (has_storage and self.build_cache_enabled):
--> 803 build_task(self.get_engine_dir())
805 return self.get_engine_dir()

File /usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm_utils.py:749, in CachedModelLoader._build_model..build_task(engine_dir)
747 assert self.mpi_session
748 # The engine_dir:Path will be stored to MPINodeState.state
--> 749 build_infos = self.mpi_session.submit_sync(
750 CachedModelLoader._node_build_task,
751 engine_dir=engine_dir,
752 **model_loader_kwargs)
753 self.llm_build_stats.build_steps_info = build_infos[0]
755 else: # single-gpu

File /usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/mpi_session.py:156, in MpiPoolSession.submit_sync(self, task, *args, **kwargs)
151 def submit_sync(self, task: Callable[..., T], *args, **kwargs) -> List[T]:
152 futures = [
153 self.mpi_pool.submit(task, *args, **kwargs)
154 for i in range(self.n_workers)
155 ]
--> 156 return [future.result() for future in futures]

File /usr/lib/python3.12/concurrent/futures/_base.py:456, in Future.result(self, timeout)
454 raise CancelledError()
455 elif self._state == FINISHED:
--> 456 return self.__get_result()
457 else:
458 raise TimeoutError()

File /usr/lib/python3.12/concurrent/futures/_base.py:401, in Future.__get_result(self)
399 if self._exception:
400 try:
--> 401 raise self._exception
402 finally:
403 # Break a reference cycle with the exception in self._exception
404 self = None

AssertionError: Must be np.ndarray. Proper usage: get parameter.raw_value before getting parameter.value

additional notes

Please let me know how I may load up the model, I need to do this urgently to proceed with my work. I tried tensor_parallel_size=4 as well, but that gave me a memory error, could not understand why. I will post that issue too.

I was unable to use the auto_parallel feature for any model, though the tensor_parallel_size feature worked on some models. The image that I used is nvcr.io/nvidia/tensorrt-llm/release:0.20.0

Reproduce the reported LLM construction with the TensorRT-LLM 0.20.0 container and the Llama-3.3-70B-Instruct-FP8 model. Start at quantization/layers.py line 1433, then inspect parameter.py and the call path through models/llama/model.py; done means auto_parallel initialization completes without the raw_value assertion.