Direct export/build fails for Qwen3 VL FP8 fine-grained 2D block-FP8 checkpoints

Describe the bug

I am trying to directly export and build the publicly available pre-quantized Hugging Face checkpoint:

Qwen/Qwen3-VL-4B-Instruct-FP8

with TensorRT Edge-LLM.

With TensorRT Edge-LLM 0.10.1, this checkpoint is not recognized and handled correctly.

The model uses Qwen's fine-grained 2D block-FP8 quantization format. Its config.json contains:

{
  "quant_method": "fp8",
  "activation_scheme": "dynamic",
  "weight_block_size": [128, 128]
}

The linear weights and scales are stored as:

weight:
  FP8 E4M3
  shape = [N, K]

weight_scale_inv:
  FP32
  shape = [N / 128, K / 128]

For example, the first attention q_proj contains:

model.layers.0.self_attn.q_proj.weight
  dtype: torch.float8_e4m3fn
  shape: [4096, 2560]

model.layers.0.self_attn.q_proj.weight_scale_inv
  dtype: torch.float32
  shape: [32, 20]

The shapes correspond exactly to a 2D 128 x 128 block layout:

4096 / 128 = 32
2560 / 128 = 20

So each 128 x 128 FP8 weight block has one FP32 dequantization scale.

Observed behavior

In TensorRT Edge-LLM 0.10.1, this quantization format is not recognized as a separate 2D block-FP8 format.

During model construction, the affected linear layers are created as FP16 linear layers, while the checkpoint loader preserves the original FP8 checkpoint weights.

This results in a computation path equivalent to:

FP16 activation
       ×
FP8 weight
       ↓
MatMul

During TensorRT LLM engine build, this fails with:

IMatrixMultiplyLayer must have same input types.
A is of type Half and B is of type FP8.

For example, the issue appears in projections such as:

model.layers.0.self_attn.q_proj

Therefore, the issue is not that TensorRT has no FP8 support. The problem is that the Qwen checkpoint's:

128 x 128 fine-grained 2D block-FP8

format is not correctly recognized and lowered by this export path.

Quantization format analysis

The public Qwen checkpoint uses:

weight_block_size = [128, 128]
weight_scale_inv  = [out_blocks, in_blocks]

This is different from the Qwen3-VL FP8 path commonly exercised by the existing TensorRT Edge-LLM CI, where a base model is first quantized through the TensorRT Edge-LLM / ModelOpt quantization pipeline and then exported.

The case here is specifically:

an externally pre-quantized fine-grained FP8 checkpoint

being passed directly to TensorRT Edge-LLM.

Observation on current main

I also inspected the current main branch and noticed that the newer experimental direct builder already contains dedicated 2D block-FP8 support:

QUANT_FP8_BLOCK

experimental/builder/ops/backend.py::fp8_block_linear()

That implementation directly uses TensorRT IDequantizeLayer with a 2D block shape.

Conceptually, the path is:

FP8 weight [N, K]
        +
FP32 2D block scales
[out_blocks, in_blocks]
        ↓
TensorRT IDequantizeLayer
block_shape = (block_n, block_k)
        ↓
FP16 weight [N, K]
        ↓
MatMul

For example, for:

weight = [4096, 2560]
scale grid = [32, 20]

the block shape is:

block_shape = (128, 128)

This indicates that current main already has the TensorRT backend capability required to represent this type of 2D block-FP8 quantization.

However, the current direct-builder checkpoint contract appears to expect a scale tensor named:

.weight_scale

with an internal block-scale layout similar to:

[out_blocks, 1, in_blocks, 1]

while the public Qwen checkpoint stores:

.weight_scale_inv

[out_blocks, in_blocks]

The Qwen checkpoint also describes the quantization format through:

{
  "quant_method": "fp8",
  "weight_block_size": [128, 128]
}

rather than through the existing FP8_PB-style checkpoint contract.

Therefore, from the current code structure, the TensorRT native block-FP8 backend already appears to exist, but there is still a missing checkpoint parsing / adaptation layer between:

public Qwen fine-grained FP8 checkpoint
        ↓
existing QUANT_FP8_BLOCK / fp8_block_linear()

Local prototype and validation

To verify the root cause, I implemented a local prototype on the TensorRT Edge-LLM 0.10.1 legacy ONNX export path.

The prototype includes:

1. A dedicated QUANT_FP8_BLOCK quantization type

2. Detection of:
   quant_method = fp8
   weight_block_size = [128, 128]

3. A dedicated FP8BlockLinear implementation

4. Loading support for:
   weight
   weight_scale_inv

5. Offline 2D block weight repacking before ONNX export

6. ONNX DequantizeLinear lowering for block-FP8 weights

Because the legacy ONNX path cannot directly set:

IDequantizeLayer.block_shape = (128, 128)

I repacked the 2D block-FP8 weights offline before ONNX export.

The original layout:

[N, K]

is transformed as:

[Nb, 128, Kb, 128]
        ↓
[Nb, Kb, 128, 128]
        ↓
[Nb * Kb, 16384]

For example, for q_proj:

original:

weight:
[4096, 2560]

scale:
[32, 20]

after offline repacking:

weight:
[640, 16384]

scale:
[640]

where:

640   = 32 × 20
16384 = 128 × 128

The resulting ONNX graph becomes:

FP8 initializer
       ↓
DequantizeLinear
       ↓
FP16
       ↓
restore original [N, K] layout
       ↓
MatMul

Why the block repack must happen before ONNX export

I initially tried to express the block layout conversion inside the ONNX graph:

FP8 Constant
    ↓
Reshape
    ↓
Transpose
    ↓
Reshape
    ↓
DequantizeLinear

This is mathematically correct and removes the original:

FP16 × FP8 MatMul

problem.

However, TensorRT later rejects the graph with:

Quantized constant (...) is only allowed before DQ or PLUGIN_V2 or PLUGIN_V3 node.

This means that for a TensorRT FP8 quantized constant, the direct consumer must be a dequantization layer or an appropriate plugin.

Therefore, the working legacy ONNX path must be:

offline block repack
       ↓
FP8 Constant
       ↓
DequantizeLinear
       ↓
FP16
       ↓
Reshape / Transpose
       ↓
MatMul

ONNX validation after the prototype change

For Qwen3-VL-4B-Instruct-FP8, the exported LLM ONNX graph contains:

FP8 initializers: 252
DequantizeLinear: 252

I also checked the direct consumer of every FP8 initializer:

PASS:
every FP8 initializer is consumed directly
by exactly one DequantizeLinear

For example, the first q_proj becomes:

q_proj.weight
FP8 initializer
       ↓
DequantizeLinear
       ↓
FP16
       ↓
Reshape / Transpose / Reshape
       ↓
MatMul

Final validation result

With the prototype changes:

LLM ONNX export                 PASS
FP8 graph topology validation   PASS
TensorRT LLM engine build       PASS
TensorRT Visual engine build    PASS
Qwen3-VL end-to-end inference   PASS

The TensorRT LLM engine builds successfully:

[TP rank 0/1] LLM engine built successfully.

End-to-end Qwen3-VL multimodal inference also completes successfully.

Suggested direction for current main

For current main, I do not think the legacy ONNX offline-repacking workaround needs to be ported directly.

Since current main already contains:

QUANT_FP8_BLOCK
+
fp8_block_linear()
+
TensorRT native IDequantizeLayer.block_shape

a cleaner implementation would likely be:

Public Qwen FP8 checkpoint

weight:
[N, K] FP8

weight_scale_inv:
[Nb, Kb] FP32

quantization_config:
weight_block_size = [128, 128]

        ↓

Checkpoint parsing / adaptation

        ↓

QUANT_FP8_BLOCK

        ↓

Existing fp8_block_linear()

        ↓

TensorRT IDequantizeLayer
block_shape = (128, 128)

        ↓

FP16 MatMul

The missing adaptation could potentially include:

1. Recognize:
   quant_method = "fp8"
   + weight_block_size = [128, 128]

2. Map the public Qwen checkpoint tensor:
   .weight_scale_inv

   to the internal block-scale representation used by the builder.

3. Adapt the Qwen scale shape:
   [Nb, Kb]

   to the existing internal block-scale layout expected by the backend.

This would allow the public Qwen checkpoint to reuse the existing native block-FP8 TensorRT backend.

If this direction is aligned with the intended architecture, I would be happy to clean up the implementation and contribute a PR.

Steps/Code to reproduce bug

Use the model:

Qwen/Qwen3-VL-4B-Instruct-FP8

The checkpoint contains:

quant_method = fp8
weight_block_size = [128, 128]
weight_scale_inv

First, export the model directly.

Installation method:

TensorRT Edge-LLM was built from source, with the Python package installed in editable mode.

TensorRT Edge-LLM: 0.10.1

Export command used:

export MODEL_DIR=/workspace/models/Qwen3-VL-4B-Instruct-FP8
export MODEL_WORKSPACE=/workspace/trtllm_workspace/Qwen3-VL-4B-Instruct-FP8

export ONNX_DIR=$MODEL_WORKSPACE/onnx
export ENGINE_DIR=$MODEL_WORKSPACE/engines

tensorrt-edgellm-export \
    "$MODEL_DIR" \
    "$ONNX_DIR"

The ONNX export itself completes successfully.

Then build the LLM engine:

./build/examples/llm/llm_build \
    --onnxDir "$ONNX_DIR/llm" \
    --engineDir "$ENGINE_DIR/llm" \
    --maxBatchSize 1 \
    --maxInputLen 7168 \
    --maxKVCacheCapacity 8192

With the unmodified 0.10.1 path, the build fails on the FP8 linear weight path with an error such as:

IMatrixMultiplyLayer must have same input types.
A is of type Half and B is of type FP8.

Installation method:

Export command used:

# Paste the exact command(s) you ran, for example:
# tensorrt-edgellm-quantize-llm --model_dir Qwen/Qwen3-0.6B --output_dir ./quantized/qwen3-0.6b --quantization fp8
# tensorrt-edgellm-export-llm --model_dir ./quantized/qwen3-0.6b --output_dir ./onnx_models/qwen3-0.6b

Expected behavior

TensorRT Edge-LLM should recognize public Qwen checkpoints using:

quant_method = fp8
weight_block_size = [128, 128]
weight_scale_inv

as fine-grained 2D block-FP8 checkpoints and map them to the existing 2D block-FP8 TensorRT implementation.

Ideally:

Public Qwen checkpoint
        ↓
detect 128 × 128 block FP8
        ↓
QUANT_FP8_BLOCK
        ↓
existing fp8_block_linear()
        ↓
TensorRT IDequantizeLayer
block_shape = (128, 128)
        ↓
FP16 weight
        ↓
MatMul

If direct support for this externally pre-quantized checkpoint format is not intended, the exporter should at least detect it explicitly and report an unsupported quantization format instead of constructing an FP16 linear layer containing an FP8 checkpoint weight and failing later during TensorRT engine build.

System information (x86 Host with GPU)

Note: this issue template mentions that the Python export pipeline normally requires an x86-64 host. My reproduction was performed directly on a Jetson AGX Thor (aarch64).

In this environment, ONNX export completed successfully. The original unmodified path failed during the subsequent TensorRT LLM engine build.

• Container used (if applicable): Docker container running on NVIDIA Jetson AGX Thor
• OS: Ubuntu 24.04.4 LTS (Noble Numbat)
• CPU architecture: aarch64
• GPU name: NVIDIA Jetson AGX Thor
• GPU memory size: Unified-memory platform
• Number of GPUs: 1
• Library versions:
  • Python: 3.12.3
  • TensorRT Edge-LLM version or commit hash: 0.10.1
  • CUDA: 13.2
  • TensorRT: 10.16.1.11
  • PyTorch: 2.12.0a0+0291f960b6.nv26.04.48445190
  • Transformers: 5.17.0
  • ModelOpt: 0.42.0
  • ONNX: 1.19.0
• Any other details that may help:
  • Platform: NVIDIA Jetson AGX Thor Developer Kit
  • GPU architecture / compute capability: SM110
  • Model: Qwen/Qwen3-VL-4B-Instruct-FP8
  • Weight dtype: FP8 E4M3
  • Block size: 128 × 128
  • Scale tensor used by the public checkpoint: weight_scale_inv

import platform
import re
import subprocess


def get_nvidia_gpu_info():
    try:
        nvidia_smi = (
            subprocess.check_output(
                "nvidia-smi --query-gpu=name,memory.total --format=csv,noheader,nounits",
                shell=True,
            )
            .decode("utf-8")
            .strip()
            .split("\n")
        )
        if len(nvidia_smi) > 0:
            gpu_name = nvidia_smi[0].split(",")[0].strip()
            gpu_memory = round(float(nvidia_smi[0].split(",")[1].strip()) / 1024, 1)
            gpu_count = len(nvidia_smi)
            return gpu_name, f"{gpu_memory} GB", gpu_count
    except Exception:
        return "?", "?", "?"


def get_cuda_version():
    try:
        nvcc_output = subprocess.check_output("nvcc --version", shell=True).decode("utf-8")
        match = re.search(r"release (\d+\.\d+)", nvcc_output)
        if match:
            return match.group(1)
    except Exception:
        return "?"


def get_package_version(package):
    try:
        return getattr(__import__(package), "__version__", "?")
    except Exception:
        return "?"


def get_tensorrt_edgellm_version():
    try:
        import tensorrt_edgellm
        return tensorrt_edgellm.__version__
    except Exception:
        return "?"


# Get system info
os_info = f"{platform.system()} {platform.release()}"
if platform.system() == "Linux":
    try:
        os_info = (
            subprocess.check_output("cat /etc/os-release | grep PRETTY_NAME | cut -d= -f2", shell=True)
            .decode("utf-8")
            .strip()
            .strip('"')
        )
    except Exception:
        pass

cpu_arch = platform.machine()
gpu_name, gpu_memory, gpu_count = get_nvidia_gpu_info()
cuda_version = get_cuda_version()

# Print system information in the format required for the issue template
print("=" * 70)
print("## System information (x86 Host with GPU)")
print()
print("- Container used (if applicable): " + "?")
print("- OS (e.g., Ubuntu 22.04, CentOS 7): " + os_info)
print("- CPU architecture: " + cpu_arch)
print("- GPU name (e.g. H100, A100, RTX 4090): " + gpu_name)
print("- GPU memory size: " + gpu_memory)
print("- Number of GPUs: " + str(gpu_count))
print("- Library versions:")
print("  - Python: " + platform.python_version())
print("  - TensorRT Edge-LLM version or commit hash: " + get_tensorrt_edgellm_version())
print("  - CUDA: " + cuda_version)
print("  - PyTorch: " + get_package_version("torch"))
print("  - Transformers: " + get_package_version("transformers"))
print("  - ModelOpt: " + get_package_version("modelopt"))
print("  - ONNX: " + get_package_version("onnx"))
print("- Any other details that may help: " + "?")
print("=" * 70)

Start by tracing how the Qwen quantization config is parsed into QUANT_FP8_BLOCK and compare it with experimental/builder/ops/backend.py::fp8_block_linear(). Reproduce the failure with the export command and llm_build command from the issue. Done means the checkpoint is recognized without an FP16 × FP8 mismatch and the LLM and visual engine builds succeed.