NVIDIA / NVIDIA/cuvs

[CI] SCANN Test flaky

Open
#2,549 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Noticed that NEIGHBORS_ANN_SCANN_TEST failed:
Full logs here: https://github.com/NVIDIA/cuvs/actions/runs/33689912368/job/100455474437?pr=2264
Env:

  • RTX Pro 6000
  • CUDA 13.3.0,
  • python 3.13,
  • amd64
  • ubuntu26.04
  • latest-driver, latest-deps
[ RUN      ] ScaNN/f32_i64.build_host_input/13
/tmp/conda-bld-output/bld/rattler-build_libcuvs-headers/work/cpp/tests/neighbors/ann_scann/../ann_scann.cuh:267: Failure
Expected: (max_error) < (max_allowed_error * 1.5f), actual: 1.43570065 vs 1.425
Max reconstruction error too large: 1.4357006549835205

[  FAILED  ] ScaNN/f32_i64.build_host_input/13, where GetParam() = 80-byte object <00-10 00-00 08-00 00-00 00-00 00-00 00-00 00-40 20-00 00-00 00-00 00-00 00-10 00-00 00-00 00-00 18-00 00-00 00-00 80-3F 00-00 80-3F 02-00 00-00 08-00 00-00 00-00 00-00 00-10 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 00-00 C0-7F 00-00 00-00> (61 ms)

September 9th:

https://github.com/NVIDIA/cuvs/pull/2565#issuecomment-5612135083

[ RUN      ] ScaNN/f32_i64.build_host_input/12
/tmp/conda-bld-output/bld/rattler-build_libcuvs-headers/work/cpp/tests/neighbors/ann_scann/../ann_scann.cuh:267: Failure
Expected: (max_error) < (max_allowed_error * 1.5f), actual: 1.43224525 vs 1.425
Max reconstruction error too large: 1.4322452545166016

[  FAILED  ] ScaNN/f32_i64.build_host_input/12, where GetParam() = 80-byte object <00-10 00-00 06-00 00-00 00-00 00-00 00-00 00-40 20-00 00-00 00-00 00-00 00-10 00-00 00-00 00-00 18-00 00-00 00-00 80-3F 00-00 80-3F 02-00 00-00 08-00 00-00 00-00 00-00 00-10 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 00-00 C0-7F 00-00 00-00> (56 ms)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with cpp/tests/neighbors/ann_scann/ann_scann.cuh around line 267 and run the NEIGHBORS_ANN_SCANN_TEST using the reported environment. Compare the failing build_host_input cases and the logged max_error against max_allowed_error. Done means the test no longer flakes or the failure has a demonstrated, stable explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ci-cd, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.