microsoft / microsoft/onnxruntime

MLAS AVX2 M=1 CompInt8 SQNBit GEMM produces wrong results for asymmetric (zero-point) 4-bit weights

Open
#29,853 1 comment 1 reaction 2 assignees Claimed by @tianleiwu View on GitHub
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

MLAS's AVX2 `M=1` `CompInt8` SQNBit (blockwise 4-bit) GEMM produces numerically incorrect results for **asymmetric** (non-null zero-point) 4-bit weights, on hosts without AVX-512.

**Affected kernel:** `SQ4BitGemmM1Kernel_CompInt8_avx2` (`sqnbitgemm_kernel_avx2.cpp`) dispatching to `MlasQ4Int8GemmM1KernelBlkLen32Avx2` and the `BlkLen>=64` equivalent (`sqnbitgemm_kernel_avx2_int8_blklen32.h` / `sqnbitgemm_kernel_avx_common.h`, `load_and_mul_sum_s8_quads_with_zp_avx2`).

On an AVX2 host without AVX-512, `MlasQNBitGemmBatch` with `BlkBitWidth=4`, `ComputeType=CompInt8`, `CountM==1`, and a non-null `QuantBZeroPoint` (asymmetric quantization) returns wrong output for every block length (16/32/64/128). The corresponding AVX-512 kernel, and the AVX2 kernel for `CountM>1`, are correct.

### Expected behavior

Result matches the dequantized-weight reference `sum_k a_k * scale_b * (qb_k - zp)`, within int8 activation-quantization tolerance (~2e-2 relative), as it does on AVX-512 and for `CountM>1` on AVX2.

### Actual behavior

~46% error. Minimal observed case: `M=1, N=96, K=256, BlkLen=32, asymmetric, no bias` → `mlas = 6.0945644` vs `reference = 11.28814` (abs diff 5.19).

### Reproduction (Intel SDE 9.48.0)

- `sde64 -hsw -- ` (Haswell / AVX2): **FAILS** for M=1 asymmetric, BlkLen 32/64/128.
- `sde64 -skx -- ` (Skylake-X / AVX-512): **PASSES**.
- Same inputs, same packed B + zero-point buffer; only the runtime-dispatched compute kernel differs, so the defect is isolated to the AVX2 M=1 zero-point path.
- Symmetric (`QuantBZeroPoint == nullptr`) is correct on AVX2.
- Asymmetric `CountM>1` is correct on AVX2 (it applies the zero point through the precomputed `QuantBBlkSum`/`ABlockSum` block-sum correction; the M=1 path instead subtracts the zero point per element inside the dot product, which is where the divergence appears).

**Suspected area:** the M=1 asymmetric accumulation (`accumulate_mul_sum_avx2` → `load_and_mul_sum_s8_quads_with_zp_avx2`, `get_zp` half-byte selection / `_mm256_maddubs_epi16` sign handling) disagrees with both the AVX-512 M=1 path and the AVX2 M>1 block-sum path.

### Impact

Any consumer calling MLAS SQNBit `CompInt8` directly for single-row (M=1 / GEMV / decode) asymmetric 4-bit weights on AVX2-only hardware gets wrong numerics. Symmetric weights and AVX-512 hardware are unaffected.

### Environment

x86-64 Linux, AVX2 (no AVX-512), MLAS from ONNX Runtime `main`; reproduced deterministically under Intel SDE `-hsw`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.