NVIDIA / NVIDIA/cuvs

[BUG] Flaky java test `CagraRandomizedIT.testResultsTopKWithRandomValues`

Open
#1,048 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Java
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

It appears that CagraRandomizedIT.testResultsTopKWithRandomValues is flaky, in that there are occasional failures. I ran into this as part of #1003:

[TEST-CagraRandomizedIT.testResultsTopKWithRandomValues-seed#[223F9EFF00E0082E]] INFO com.nvidia.cuvs.CuVSTestCase - Results returned for query 7: [146, 42, 31, 105, 145, 149, 48, 17, 15, 192, 127, 46, 65, 154, 71, 67, 133, 39, 151, 101, 35, 25, 190, 77, 131, 86, 158, 7, 174, 191, 49]
[TEST-CagraRandomizedIT.testResultsTopKWithRandomValues-seed#[223F9EFF00E0082E]] INFO com.nvidia.cuvs.CuVSTestCase - Expected results for query 7: [146, 42, 31, 105, 145, 149, 48, 17, 15, 127, 46, 65, 154, 71, 67, 133, 49, 187, 178, 177, 39, 78, 151, 101, 35, 54, 25, 172, 190, 77, 53, 131, 86, 158]
Error:  Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.84 s <<< FAILURE! -- in com.nvidia.cuvs.CagraRandomizedIT
Error:  com.nvidia.cuvs.CagraRandomizedIT.testResultsTopKWithRandomValues -- Time elapsed: 25.84 s <<< FAILURE!
java.lang.AssertionError: Not found in expected list: 192
	at __randomizedtesting.SeedInfo.seed([223F9EFF00E0082E:B1342675EB3F66E]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at com.nvidia.cuvs@25.08.0/com.nvidia.cuvs.CuVSTestCase.compareResults(CuVSTestCase.java:103)
	at com.nvidia.cuvs@25.08.0/com.nvidia.cuvs.CagraRandomizedIT.tmpResultsTopKWithRandomValues(CagraRandomizedIT.java:155)
	at com.nvidia.cuvs@25.08.0/com.nvidia.cuvs.CagraRandomizedIT.testResultsTopKWithRandomValues(CagraRandomizedIT.java:50)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:426)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)

This seems to be with random-seed 223F9EFF00E0082E.

It would be good to fix this failure, so that CI is more predictable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CagraRandomizedIT.java around lines 50 and 155, using random seed 223F9EFF00E0082E to reproduce the failure. Read CuVSTestCase.java:103 to understand how the result comparison reports the mismatch. The work is done when the test no longer fails intermittently and passes reliably in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.