NVIDIA / NVIDIA/Model-Optimizer
Quantizing error for Qwen3.6 35B A3B
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 604
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 142
Description
Describe the bug
Qwen3.6-35B A3B NVPF4 Quantization IndexError: tuple index out of range error
Steps/Code to reproduce bug
docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v "./output_models:/workspace/output_models" -v "$HOME/.cache/huggingface:/root/.cache/huggi
ngface" -e HF_TOKEN=$HF_TOKEN nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20 bash -c "
git clone --depth 1 --branch main https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
cd /app/Model-Optimizer/ && \
pip install -e '.' && \
cd /app/Model-Optimizer/examples/hf_ptq && \
export ROOT_SAVE_PATH='/workspace/output_models' && \
scripts/huggingface_example.sh --model 'Qwen/Qwen3.6-35B-A3B' --quant nvfp4 --tasks quant --tp 1
"
Loading safetensors weights in parallel: 100%|██████████| 3/3 [00:02<00:00, 1.18it/s]
[07/07/2026-03:31:36] [TRT-LLM] [E] [executor] Failed to initialize executor on rank 0: tuple index out of range
[07/07/2026-03:31:36] [TRT-LLM] [E] [executor] Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 516, in create_py_executor
model_engine = PyTorchModelEngine(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 341, in __init__
self.model, moe_load_balancer = self.model_loader.load(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 519, in load
self._call_load_weights(model.load_weights, weights,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 1135, in _call_load_weights
load_method(weights, **kargs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen3_next.py", line 980, in load_weights
new_weights = weight_mapper.preprocess_weights(weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 327, in preprocess_weights
packed_weights = self._pack_split_projections(normalized_weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 258, in _pack_split_projections
q_tensor, k_tensor, v_tensor = split_packed_qkv(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 169, in _split_qkv_tensor
assert tensor.shape[0] == expected_total, (
~~~~~~~~~~~~^^^
IndexError: tuple index out of range
[07/07/2026-03:31:36] [TRT-LLM] [E] [executor] Executor worker initialization error: Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 516, in create_py_executor
model_engine = PyTorchModelEngine(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 341, in __init__
self.model, moe_load_balancer = self.model_loader.load(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 519, in load
self._call_load_weights(model.load_weights, weights,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 1135, in _call_load_weights
load_method(weights, **kargs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen3_next.py", line 980, in load_weights
new_weights = weight_mapper.preprocess_weights(weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 327, in preprocess_weights
packed_weights = self._pack_split_projections(normalized_weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 258, in _pack_split_projections
q_tensor, k_tensor, v_tensor = split_packed_qkv(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 169, in _split_qkv_tensor
assert tensor.shape[0] == expected_total, (
~~~~~~~~~~~~^^^
IndexError: tuple index out of range
IndexError: tuple index out of range
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/Model-Optimizer/examples/hf_ptq/run_tensorrt_llm.py", line 96, in <module>
run(args)
File "/app/Model-Optimizer/examples/hf_ptq/run_tensorrt_llm.py", line 71, in run
llm = LLM(
^^^^
File "/app/Model-Optimizer/modelopt/deploy/llm/generate.py", line 148, in __init__
super().__init__(
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1877, in __init__
super().__init__(model, tokenizer, tokenizer_mode, skip_tokenizer_init,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1717, in __init__
super().__init__(model,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 349, in __init__
self._build_model()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1824, in _build_model
self._executor = self._executor_cls.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^ 22:42:35 [196/2078]
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 639, in create
return GenerationExecutor._create_ipc_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 510, in _create_ipc_executor
return GenerationExecutorProxy(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 150, in __init__
self._start_executor_workers(worker_kwargs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 437, in _start_executor_workers
raise RuntimeError(
RuntimeError: Executor worker returned error
Expected behavior
Working quantized model output
System information
- Container used: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20
- OS: Ubuntu 22.04,
- CPU architecture: aarch64
- GPU: GB10
- GPU memory size: 128GB
- Number of GPUs: 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the Qwen/Qwen3.6-35B-A3B NVFP4 command from the issue and follow the traceback into modeling_qwen3_next.py, qwen3_5_weight_mapper.py, and _split_qkv_tensor. Inspect how preprocess_weights handles the failing tensor; done means quantization completes without the tuple index error and the model loads successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100