microsoft / microsoft/onnxruntime
[MLAS] Validating x86 kernels that no CI machine can reach, and a cheap way to keep doing it
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
While working on the CompInt8 kernels I wanted to check the x86 paths that my machine cannot execute, so I ran the existing MLAS unit tests under Intel SDE with different CPUID targets, and used SDE's instruction histogram to confirm the intended kernel actually retired instead of a fallback. All of this is against current main.
| kernel | ISA gate | reached with | result | instructions retired |
|---|---|---|---|---|
MlasGemmS8S8KernelAvx2Vnni, MlasGemmS8U8KernelAvx2Vnni, MlasGemmU8U8DispatchAvx2Vnni |
AVX-VNNI-INT8 | SDE -srf |
640 pass per int32 suite, 120 per fp32 suite | VPDPBSSD, VPDPBSUD, VPDPBUUD, about 1.8 billion each |
MlasGemmU8S8DispatchAmx |
AMX-TILE and AMX-INT8 | SDE -spr |
640 pass | TDPBUSD 3675545, TILELOADD 3969423 |
MlasCastF16ToF32KernelAvx |
AVX-NE-CONVERT | SDE -srf |
26 pass | vcvtneeph2ps and vcvtneoph2ps, 4220 each |
Everything passed, so this is not a bug report. Two things came out of it that seemed worth writing up.
The first is which machines can run this code. AVX-VNNI-INT8 and AVX-NE-CONVERT are client and E-core features, so they are not present on Skylake-SP, Cascade Lake, Ice Lake-SP, Sapphire Rapids or Granite Rapids. Unless a build agent happens to be Sierra Forest class, the five AVX-VNNI-INT8 int8 GEMM kernels and the AVX-NE-CONVERT cast kernel are never executed in CI.
The second is that a passing run does not tell you which kernel ran. The same *Cast* group gives an identical result whether the ISA kernel or the fallback was used:
| CPUID target | test result | vcvtneeph2ps plus vcvtneoph2ps retired |
|---|---|---|
-adl Alder Lake |
26 passed | none |
-gnr Granite Rapids |
26 passed | none |
-srf Sierra Forest |
26 passed | 4220 each |
The QNBitGemm tests have a related shape, since they call MlasIsQNBitGemmAvailable and MlasQNBitGemmFp16DirectQuantASupported and return early when a dispatch entry is missing, so an unwired dispatch pointer also reads as a pass. A gate that ends up false anywhere in this area is invisible to the suite, which is part of what makes changes in this area hard to review with confidence.
On cost, SDE was cheaper than I expected. The 640 test int8 GEMM suites take about 100 seconds under emulation against about 40 seconds native, so roughly 2.5x, and the cast group is 3 seconds. AMX needed no special setup beyond the -spr flag. SDE itself is free and needs no particular hardware.
If this is useful I am happy to contribute it in whatever form suits you, either a script and a pipeline definition for an opt in job, or just a documented developer side recipe under the MLAS test directory. The histogram check is also useful when reviewing a kernel PR, since it answers whether the kernel under review was reached at all.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the MLAS test directory, reviewing the int8 GEMM, cast, and QNBitGemm tests plus MlasIsQNBitGemmAvailable and MlasQNBitGemmFp16DirectQuantASupported. Compare the existing test invocation with Intel SDE CPUID targets and histogram checks. Done means an opt-in validation job or documented recipe can exercise unavailable x86 kernels and confirm the intended instructions retired.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ci-cd, machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100