XnnpackBackend init failed while running export-based quantized pte model using ExecuTorch Runtime

🐛 Describe the bug

I am following the tutorial of the exporting quantized model using executorch, specifically both export-based and source-based quantization. The source-based model runs without any error but a warning [program.cpp:135] InternalConsistency verification requested but not available. But the export-based code breaks when I load the dumped, quantized ExecuTorch module into the ExecuTorch Runtime. The error indicates that the XNNPack backend is not correctly initialized.

Export-based quantization:

import torch

from torch.export import export_for_training, export
from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
from executorch.exir import to_edge_transform_and_lower, EdgeCompileConfig
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import XNNPACKQuantizer, get_symmetric_quantization_config
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from executorch.runtime import Verification, Runtime, Program, Method


# define calibration function
def calibrate(model, data_loader):
    with torch.no_grad():
        for image, target in data_loader:
            model(image)

m = torch.nn.Sequential(torch.nn.Linear(10, 10))
example_inputs = (torch.randn(2, 10), )
print(m(example_inputs[0]))

m = export_for_training(m, example_inputs, strict=False).module()
quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config(is_dynamic=True))
m = prepare_pt2e(m, quantizer)
calibrate(m, example_inputs)
m = convert_pt2e(m)
m = export(m, example_inputs)

et = to_edge_transform_and_lower(
    m,
    compile_config=EdgeCompileConfig(_check_ir_validity=False),
    partitioner=[XnnpackPartitioner()],
).to_executorch()
    
print('executorch direct execution')
print(et.exported_program().module()(*example_inputs))

et_runtime: Runtime = Runtime.get()
program: Program = et_runtime.load_program(et.buffer)
print("Program methods:", program.method_names)
forward: Method = program.load_method("forward")

outputs = forward.execute(example_inputs)
print(f"Ran forward({example_inputs[0].shape})")
print(f"  outputs: {outputs[0].shape}")

Source-based quantization code with torchao quantizers:

import torch
from torch.export import export
from executorch.exir import to_edge_transform_and_lower, EdgeCompileConfig
from executorch.runtime import Runtime, Program, Method
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from torchao.quantization import quantize_, int8_weight_only

    
example_inputs = (torch.randn(2, 10), )
print("float model")
m = torch.nn.Sequential(torch.nn.Linear(10, 10)).eval()
print(m)
print(m(*example_inputs))

quantize_(m, int8_weight_only())
m = export(m, example_inputs)
et = to_edge_transform_and_lower(
    m,
    compile_config=EdgeCompileConfig(_check_ir_validity=False),
    partitioner=[XnnpackPartitioner()],
).to_executorch()
    
print('executorch direct execution')
print(et.exported_program().module()(*example_inputs))

print("executorch runtime")
et_runtime: Runtime = Runtime.get()
program: Program = et_runtime.load_program(et.buffer)
print("Program methods:", program.method_names)
forward: Method = program.load_method("forward")

outputs = forward.execute(example_inputs)
print(f"Ran forward({example_inputs[0].shape})")
print(f"  outputs: {outputs[0].shape}")

The error when running the export-based quantization:

[program.cpp:135] InternalConsistency verification requested but not available
[XNNCompiler.cpp:750] Failed to create linear node 10, with code: xnn_status_invalid_parameter
[XNNPACKBackend.cpp:105] XNNCompiler::compileModel failed: 0x1
[method.cpp:109] Init failed for backend XnnpackBackend: 0x1
Traceback (most recent call last):
  File "/nfs/home/zizhang/workspace/convert_litert/q_export_based_issue.py", line 38, in <module>
    program: Program = et_runtime.load_program(et.buffer)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/disk1/python311-executorch-1eb2f94/lib64/python3.11/site-packages/executorch/runtime/__init__.py", line 213, in load_program
    m = self._legacy_module._load_for_executorch_from_buffer(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: loading method forward failed with error 0x1

---

Initially, both code snippets result in this XNNpackBackend initialization failed error. Right before setting up this issue, I ran into this one relating to the XNNPack backend and realized I might need to tweak the CMakeList.txt a little bit. Therefore I did the following modifications:

option(EXECUTORCH_BUILD_KERNELS_OPTIMIZED "Build the optimized kernels" ON)
option(EXECUTORCH_BUILD_KERNELS_QUANTIZED "Build the quantized kernels" ON)
option(EXECUTORCH_BUILD_XNNPACK "Build the XNNPACK backend" ON)

According to the install_executorch.py, the last EXECUTORCH_BUILD_XNNPACK flag should be set automatically to ON if one installs the ExecuTorch using

./install_executorch.sh --pybind xnnpack

or even simply ./install_executorch.sh and the xnnpack pybind is introduced by default. So, that should not be the only thing that enables source-based quantization. So maybe I did it right by setting EXECUTORCH_BUILD_KERNELS_QUANTIZED to ON? If so, why does the export-based quantization, according to the tutorial, not work?

---

Another question regards the calibration step of the export-based quantization. In my code, I only want to use the dynamic quantization (int8 weight only). In this case, no calibration should be needed. But when I comment out the calibration, the output before and after the quantization are way different. But the source-based quantization doesn't suffer with this problem. Could you please maybe share your idea of why this happens?

For your better information, here are the results before and after the source-based quantization:

float model:
Sequential(
  (0): Linear(in_features=10, out_features=10, bias=True)
)
tensor([[-0.0358, -0.5517,  0.2567,  0.7979,  0.9086, -0.9824, -0.8816,  0.7467,
          0.3425, -0.3511],
        [ 0.2783,  0.4162,  0.7018, -0.3589,  0.3428, -0.4533, -0.2705,  0.6411,
         -1.0303, -0.9237]], grad_fn=<AddmmBackward0>)
=====================================
quantized model:
Sequential(
  (0): Linear(in_features=10, out_features=10, weight=AffineQuantizedTensor(shape=torch.Size([10, 10]), block_size=(1, 10), device=cpu, _layout=PlainLayout(), tensor_impl_dtype=torch.int8, quant_min=None, quant_max=None))
)
tensor([[-0.0367, -0.5518,  0.2586,  0.7983,  0.9099, -0.9846, -0.8807,  0.7512,
          0.3429, -0.3487],
        [ 0.2780,  0.4171,  0.7036, -0.3571,  0.3438, -0.4554, -0.2713,  0.6430,
         -1.0285, -0.9250]], grad_fn=<AsStridedBackward0>)

And here are the results before and after the export-based quantization:

float model:
Sequential(
  (0): Linear(in_features=10, out_features=10, bias=True)
)
tensor([[ 0.0390,  0.5013,  0.1791, -0.0260,  0.0412, -0.3811, -0.1001,  0.7664,
         -0.3072,  0.3328],
        [-0.7758,  1.0587, -0.6142, -1.4604,  0.1691, -0.4577,  1.1940,  0.3819,
         -0.0434,  0.6676]], grad_fn=<AddmmBackward0>)
/disk1/python311-executorch-1eb2f94/lib64/python3.11/site-packages/torch/ao/quantization/utils.py:408: UserWarning: must run observer before calling calculate_qparams. Returning default values.
  warnings.warn(
=====================================
quantized model:
class GraphModule(torch.nn.Module):
    def forward(self, p_getattr_l__self_____0___bias: "f32[10]", b__frozen_param0: "i8[10, 10]", input: "f32[2, 10]"):
        input_1 = input
        
         # File: /disk1/python311-executorch-1eb2f94/lib64/python3.11/site-packages/torch/nn/modules/linear.py:125 in forward, code: return F.linear(input, self.weight, self.bias)
        dequantize_per_tensor: "f32[10, 10]" = torch.ops.quantized_decomposed.dequantize_per_tensor.default(b__frozen_param0, 1.0, 0, -127, 127, torch.int8);  b__frozen_param0 = None
        choose_qparams = torch.ops.quantized_decomposed.choose_qparams.tensor(input_1, -128, 127, 0.000244140625, torch.int8)
        getitem: "f64[1]" = choose_qparams[0]
        getitem_1: "i64[1]" = choose_qparams[1];  choose_qparams = None
        quantize_per_tensor: "i8[2, 10]" = torch.ops.quantized_decomposed.quantize_per_tensor.tensor(input_1, getitem, getitem_1, -128, 127, torch.int8);  input_1 = None
        dequantize_per_tensor_1: "f32[2, 10]" = torch.ops.quantized_decomposed.dequantize_per_tensor.tensor(quantize_per_tensor, getitem, getitem_1, -128, 127, torch.int8);  quantize_per_tensor = getitem = getitem_1 = None
        linear: "f32[2, 10]" = torch.ops.aten.linear.default(dequantize_per_tensor_1, dequantize_per_tensor, p_getattr_l__self_____0___bias);  dequantize_per_tensor_1 = dequantize_per_tensor = p_getattr_l__self_____0___bias = None
        return (linear,)
        
tensor([[ 0.3101, -0.0603,  0.0957, -0.0780,  0.0196,  0.0787, -0.0511,  0.3145,
          0.0020,  0.1089],
        [ 0.3101, -0.0603,  0.0957, -0.0780,  0.0196,  0.0787, -0.0511,  0.3145,
          0.0020,  0.1089]], grad_fn=<AddmmBackward0>)

Versions

Collecting environment information...
PyTorch version: 2.7.0.dev20250131+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Oracle Linux Server 9.4 (x86_64)
GCC version: (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3.0.1)
Clang version: 14.0.6
CMake version: version 3.31.4
Libc version: glibc-2.34

Python version: 3.11.7 (main, Oct 9 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3.0.1)] (64-bit runtime)
Python platform: Linux-5.15.0-207.156.6.el9uek.x86_64-x86_64-with-glibc2.34
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 40 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
Model name: AMD EPYC 9J14 96-Core Processor
CPU family: 25
Model: 17
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
Stepping: 1
BogoMIPS: 5192.18
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves nt_good avx512_bf16 clzero xsaveerptr wbnoinvd arat npt nrip_save vgif vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor fsrm arch_capabilities
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 256 KiB (4 instances)
L1i cache: 256 KiB (4 instances)
L2 cache: 2 MiB (4 instances)
L3 cache: 16 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+dedfdaf
[pip3] numpy==2.2.3
[pip3] torch==2.7.0.dev20250131+cpu
[pip3] torchao==0.10.0+git7d879462
[pip3] torchaudio==2.6.0.dev20250131+cpu
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0.dev20250131+cpu
[pip3] triton==3.2.0
[conda] Could not collect

cc @digantdesai @mcr229 @cbilgin

Start by reproducing the export-based quantization example and the failure at Runtime.load_program, then compare it with the source-based path. Inspect XnnpackPartitioner, the XNNPACK backend initialization, and the CMakeList.txt options named in the report. Done means the exported quantized module loads and runs through ExecuTorch Runtime with outputs consistent with the direct execution path.