[BUG] Flaky java test `CagraRandomizedIT.testResultsTopKWithRandomValues`
Open
Nobody has claimed this yet.
bug
Java
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
It appears that CagraRandomizedIT.testResultsTopKWithRandomValues is flaky, in that there are occasional failures. I ran into this as part of #1003:
[TEST-CagraRandomizedIT.testResultsTopKWithRandomValues-seed#[223F9EFF00E0082E]] INFO com.nvidia.cuvs.CuVSTestCase - Results returned for query 7: [146, 42, 31, 105, 145, 149, 48, 17, 15, 192, 127, 46, 65, 154, 71, 67, 133, 39, 151, 101, 35, 25, 190, 77, 131, 86, 158, 7, 174, 191, 49]
[TEST-CagraRandomizedIT.testResultsTopKWithRandomValues-seed#[223F9EFF00E0082E]] INFO com.nvidia.cuvs.CuVSTestCase - Expected results for query 7: [146, 42, 31, 105, 145, 149, 48, 17, 15, 127, 46, 65, 154, 71, 67, 133, 49, 187, 178, 177, 39, 78, 151, 101, 35, 54, 25, 172, 190, 77, 53, 131, 86, 158]
Error: Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.84 s <<< FAILURE! -- in com.nvidia.cuvs.CagraRandomizedIT
Error: com.nvidia.cuvs.CagraRandomizedIT.testResultsTopKWithRandomValues -- Time elapsed: 25.84 s <<< FAILURE!
java.lang.AssertionError: Not found in expected list: 192
at __randomizedtesting.SeedInfo.seed([223F9EFF00E0082E:B1342675EB3F66E]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at com.nvidia.cuvs@25.08.0/com.nvidia.cuvs.CuVSTestCase.compareResults(CuVSTestCase.java:103)
at com.nvidia.cuvs@25.08.0/com.nvidia.cuvs.CagraRandomizedIT.tmpResultsTopKWithRandomValues(CagraRandomizedIT.java:155)
at com.nvidia.cuvs@25.08.0/com.nvidia.cuvs.CagraRandomizedIT.testResultsTopKWithRandomValues(CagraRandomizedIT.java:50)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:426)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
This seems to be with random-seed 223F9EFF00E0082E.
It would be good to fix this failure, so that CI is more predictable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CagraRandomizedIT.java around lines 50 and 155, using random seed 223F9EFF00E0082E to reproduce the failure. Read CuVSTestCase.java:103 to understand how the result comparison reports the mismatch. The work is done when the test no longer fails intermittently and passes reliably in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100