microsoft / microsoft/SPTAG

random test failures on older CPUs without SSE/AVX/AVX2/AVX512

Open
#316 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
5k
Forks
622
Avg merge
8h 37m
Merged PRs (30d)
5

Description

Describe the bug
There are strange random test failures on older CPUs that don't have SSE/AVX/AVX2/AVX512.

To Reproduce
Steps to reproduce the behavior:

  1. Build SPTAG on a machine without AVX/AVX2/AVX512.
  2. Run SPTAGTests
  3. See error

Expected behavior
The tests should work.

Analysis
I think that because of the -mavx2 -mavx -msse -msse2 -mavx512f -mavx512bw -mavx512dq options in the DistanceUtils target_compile_options, the compiler is generating newer instructions in the DistanceUtils library and these are not run on older CPUs. Removing the options not supported by the CPU and deleting the functions using instructions that those options enable fixes this issue. So the cause is definitely the options being enabled.

Suggestions
On Linux you can use GCC function multi-versioning to get the compiler to automatically check the CPU at runtime and dispatch to the right functions.

Screenshots
Some examples of the random failures:

[1] Start invoking BuildTrees.
[1] BKTKmeansK: 3, BKTLeafSize: 6, Samples: 100, BKTLambdaFactor:-1.000000 TreeNumber: 1, ThreadNum: 2.
unknown location(0): fatal error: in "SSDServingTest/TestHeadUInt8L2DEFAULT": memory access violation at address: 0x00000000: no mapping at fault address
./Test/src/SSDServingTest.cpp(444): last checkpoint: "TestHeadUInt8L2DEFAULT" test entry

*** 1 failure is detected in the test module "Main"
[1] Parallel TpTree Partition done
[1] Build TPTree time (s): 4
[1] Processing Tree 0 0%
unknown location(0): fatal error: in "AlgoTest/KDTTest": signal: illegal operand; address of failing instruction: 0x559a5eccc130
./Test/src/AlgoTest.cpp(22): last checkpoint

*** 1 failure is detected in the test module "Main"

Desktop:

  • OS: Debian
  • Browser None
  • Version 12 (bookworm)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the DistanceUtils target_compile_options and run SPTAGTests on a CPU without the listed SSE/AVX extensions, focusing on the failures in SSDServingTest and AlgoTest. The work is done when the test suite no longer executes unsupported instructions or crashes on older CPUs while retaining supported optimized paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
build-system, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.