microsoft / microsoft/onnxruntime
MLAS AVX2 M=1 CompInt8 SQNBit GEMM produces wrong results for asymmetric (zero-point) 4-bit weights
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
MLAS's AVX2 `M=1` `CompInt8` SQNBit (blockwise 4-bit) GEMM produces numerically incorrect results for **asymmetric** (non-null zero-point) 4-bit weights, on hosts without AVX-512.
**Affected kernel:** `SQ4BitGemmM1Kernel_CompInt8_avx2` (`sqnbitgemm_kernel_avx2.cpp`) dispatching to `MlasQ4Int8GemmM1KernelBlkLen32Avx2` and the `BlkLen>=64` equivalent (`sqnbitgemm_kernel_avx2_int8_blklen32.h` / `sqnbitgemm_kernel_avx_common.h`, `load_and_mul_sum_s8_quads_with_zp_avx2`).
On an AVX2 host without AVX-512, `MlasQNBitGemmBatch` with `BlkBitWidth=4`, `ComputeType=CompInt8`, `CountM==1`, and a non-null `QuantBZeroPoint` (asymmetric quantization) returns wrong output for every block length (16/32/64/128). The corresponding AVX-512 kernel, and the AVX2 kernel for `CountM>1`, are correct.
### Expected behavior
Result matches the dequantized-weight reference `sum_k a_k * scale_b * (qb_k - zp)`, within int8 activation-quantization tolerance (~2e-2 relative), as it does on AVX-512 and for `CountM>1` on AVX2.
### Actual behavior
~46% error. Minimal observed case: `M=1, N=96, K=256, BlkLen=32, asymmetric, no bias` → `mlas = 6.0945644` vs `reference = 11.28814` (abs diff 5.19).
### Reproduction (Intel SDE 9.48.0)
- `sde64 -hsw -- ` (Haswell / AVX2): **FAILS** for M=1 asymmetric, BlkLen 32/64/128.
- `sde64 -skx -- ` (Skylake-X / AVX-512): **PASSES**.
- Same inputs, same packed B + zero-point buffer; only the runtime-dispatched compute kernel differs, so the defect is isolated to the AVX2 M=1 zero-point path.
- Symmetric (`QuantBZeroPoint == nullptr`) is correct on AVX2.
- Asymmetric `CountM>1` is correct on AVX2 (it applies the zero point through the precomputed `QuantBBlkSum`/`ABlockSum` block-sum correction; the M=1 path instead subtracts the zero point per element inside the dot product, which is where the divergence appears).
**Suspected area:** the M=1 asymmetric accumulation (`accumulate_mul_sum_avx2` → `load_and_mul_sum_s8_quads_with_zp_avx2`, `get_zp` half-byte selection / `_mm256_maddubs_epi16` sign handling) disagrees with both the AVX-512 M=1 path and the AVX2 M>1 block-sum path.
### Impact
Any consumer calling MLAS SQNBit `CompInt8` directly for single-row (M=1 / GEMV / decode) asymmetric 4-bit weights on AVX2-only hardware gets wrong numerics. Symmetric weights and AVX-512 hardware are unaffected.
### Environment
x86-64 Linux, AVX2 (no AVX-512), MLAS from ONNX Runtime `main`; reproduced deterministically under Intel SDE `-hsw`.
Contributor guide
Assessment
This issue has not been assessed yet.