[BUG] Intermittent test failure: `test_ivf_flat.py::test_filtered_ivf_flat[0.5]`
Open
Nobody has claimed this yet.
bug
testing
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
This is to track an intermittent test failure seen in CI. For instance, the CI run for #1366:
=================================== FAILURES ===================================
_________________________ test_filtered_ivf_flat[0.5] __________________________
sparsity = 0.5
@pytest.mark.parametrize("sparsity", [0.5, 0.7, 1.0])
def test_filtered_ivf_flat(sparsity):
> run_filtered_search_test(ivf_flat, sparsity)
python/cuvs/cuvs/tests/test_ivf_flat.py:155:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
search_module = <module 'cuvs.neighbors.ivf_flat' from '/pyenv/versions/3.10.18/lib/python3.10/site-packages/cuvs/neighbors/ivf_flat/__init__.py'>
sparsity = 0.5, n_rows = 10000, n_cols = 10, n_queries = 10, k = 10
...
...
> assert recall > 0.7
E AssertionError
/pyenv/versions/3.10.18/lib/python3.10/site-packages/cuvs/tests/ann_utils.py:125: AssertionError
=========================== short test summary info ============================
FAILED python/cuvs/cuvs/tests/test_ivf_flat.py::test_filtered_ivf_flat[0.5] - AssertionError
===== 1 failed, 16008 passed, 149 skipped, 2 xfailed in 574.57s (0:09:34) ======
Apologies if this is being tracked elsewhere already.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with python/cuvs/cuvs/tests/test_ivf_flat.py, especially test_filtered_ivf_flat and its call to run_filtered_search_test. Reproduce the failure using the reported CI test and inspect the recall assertion in ann_utils.py:125. Done means identifying why the intermittent recall falls below 0.7 and making the test reliably pass without weakening its intended check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100