microsoft / microsoft/onnxruntime

[MLAS] Validating x86 kernels that no CI machine can reach, and a cheap way to keep doing it

Open
#29,862 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

While working on the CompInt8 kernels I wanted to check the x86 paths that my machine cannot execute, so I ran the existing MLAS unit tests under Intel SDE with different CPUID targets, and used SDE's instruction histogram to confirm the intended kernel actually retired instead of a fallback. All of this is against current main.

kernel ISA gate reached with result instructions retired
MlasGemmS8S8KernelAvx2Vnni, MlasGemmS8U8KernelAvx2Vnni, MlasGemmU8U8DispatchAvx2Vnni AVX-VNNI-INT8 SDE -srf 640 pass per int32 suite, 120 per fp32 suite VPDPBSSD, VPDPBSUD, VPDPBUUD, about 1.8 billion each
MlasGemmU8S8DispatchAmx AMX-TILE and AMX-INT8 SDE -spr 640 pass TDPBUSD 3675545, TILELOADD 3969423
MlasCastF16ToF32KernelAvx AVX-NE-CONVERT SDE -srf 26 pass vcvtneeph2ps and vcvtneoph2ps, 4220 each

Everything passed, so this is not a bug report. Two things came out of it that seemed worth writing up.

The first is which machines can run this code. AVX-VNNI-INT8 and AVX-NE-CONVERT are client and E-core features, so they are not present on Skylake-SP, Cascade Lake, Ice Lake-SP, Sapphire Rapids or Granite Rapids. Unless a build agent happens to be Sierra Forest class, the five AVX-VNNI-INT8 int8 GEMM kernels and the AVX-NE-CONVERT cast kernel are never executed in CI.

The second is that a passing run does not tell you which kernel ran. The same *Cast* group gives an identical result whether the ISA kernel or the fallback was used:

CPUID target test result vcvtneeph2ps plus vcvtneoph2ps retired
-adl Alder Lake 26 passed none
-gnr Granite Rapids 26 passed none
-srf Sierra Forest 26 passed 4220 each

The QNBitGemm tests have a related shape, since they call MlasIsQNBitGemmAvailable and MlasQNBitGemmFp16DirectQuantASupported and return early when a dispatch entry is missing, so an unwired dispatch pointer also reads as a pass. A gate that ends up false anywhere in this area is invisible to the suite, which is part of what makes changes in this area hard to review with confidence.

On cost, SDE was cheaper than I expected. The 640 test int8 GEMM suites take about 100 seconds under emulation against about 40 seconds native, so roughly 2.5x, and the cast group is 3 seconds. AMX needed no special setup beyond the -spr flag. SDE itself is free and needs no particular hardware.

If this is useful I am happy to contribute it in whatever form suits you, either a script and a pipeline definition for an opt in job, or just a documented developer side recipe under the MLAS test directory. The histogram check is also useful when reviewing a kernel PR, since it answers whether the kernel under review was reached at all.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the MLAS test directory, reviewing the int8 GEMM, cast, and QNBitGemm tests plus MlasIsQNBitGemmAvailable and MlasQNBitGemmFp16DirectQuantASupported. Compare the existing test invocation with Intel SDE CPUID targets and histogram checks. Done means an opt-in validation job or documented recipe can exercise unavailable x86 kernels and confirm the intended instructions retired.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ci-cd, machine-learning, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.