NVIDIA / NVIDIA/cuvs

[BUG] Possibly flaky C++ test: `SpectralClusteringTests/SpectralClusteringTestF.Result/5`

Open
#1,594 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

I ran into this with #1453. The failed CI run shows:

[ RUN      ] SpectralClusteringTests/SpectralClusteringTestF.Result/5
unknown file: Failure
C++ exception with description "exception occurred! file=/tmp/conda-bld-output/bld/rattler-build_libcuvs-headers/host_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include/raft/linalg/detail/eig.cuh line=138: eig.cuh: eigensolver couldn't converge to a solution. This usually occurs when some of the features do not vary enough.
Obtained 17 stack frames
#1 in /opt/conda/envs/test/bin/gtests/libcuvs/../../../lib/libcuvs.so: void raft::linalg::detail::eigDC<float>(raft::resources const&, float const*, unsigned long, unsigned long, float*, float*, CUstream_st*) +0xc6b [0x7c2c67c86dcb]
#2 in /opt/conda/envs/test/bin/gtests/libcuvs/../../../lib/libcuvs.so(+0x167250f) [0x7c2c6877350f]
#3 in /opt/conda/envs/test/bin/gtests/libcuvs/../../../lib/libcuvs.so(+0x167f717) [0x7c2c68780717]
#4 in /opt/conda/envs/test/bin/gtests/libcuvs/../../../lib/libcuvs.so(+0x1680bd6) [0x7c2c68781bd6]
#5 in /opt/conda/envs/test/bin/gtests/libcuvs/../../../lib/libcuvs.so: cuvs::preprocessing::spectral_embedding::transform(raft::resources const&, cuvs::preprocessing::spectral_embedding::params, raft::coo_matrix_view<float, int, int, int, true>, std::experimental::mdspan<float, std::experimental::extents<int, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_left, raft::host_device_accessor<std::experimental::default_accessor<float>, (raft::memory_type)2> >) +0x66f [0x7c2c68783dff]
#6 in /opt/conda/envs/test/bin/gtests/libcuvs/../../../lib/libcuvs.so: void cuvs::cluster::spectral::detail::fit_predict<float>(raft::resources const&, cuvs::cluster::spectral::params, raft::coo_matrix_view<float, int, int, int, true>, std::experimental::mdspan<int, std::experimental::extents<int, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<int>, (raft::memory_type)2> >) +0x237 [0x7c2c67a5b357]
#7 in /opt/conda/envs/test/bin/gtests/libcuvs/../../../lib/libcuvs.so: cuvs::cluster::spectral::fit_predict(raft::resources const&, cuvs::cluster::spectral::params, raft::coo_matrix_view<float, int, int, int, true>, std::experimental::mdspan<int, std::experimental::extents<int, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<int>, (raft::memory_type)2> >) +0x75 [0x7c2c67a59d65]
#8 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x16ac30) [0x641d8063ac30]
#9 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x192eac) [0x641d80662eac]
#10 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x193104) [0x641d80663104]
#11 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x193527) [0x641d80663527]
#12 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x1939a6) [0x641d806639a6]
#13 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x19a583) [0x641d8066a583]
#14 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x19aa71) [0x641d8066aa71]
#15 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x4decf) [0x641d8051decf]
#16 in /lib64/libc.so.6: __libc_start_main +0xe5 [0x7c2c5fbfd7e5]
#17 in ./../../..//bin/gtests/libcuvs/CLUSTER_TEST(+0x4df0e) [0x641d8051df0e]
" thrown in SetUp().

[  FAILED  ] SpectralClusteringTests/SpectralClusteringTestF.Result/5, where GetParam() = 40-byte object <88-13 00-00 64-00 00-00 08-00 00-00 08-00 00-00 28-00 00-00 05-00 00-00 9A-99 99-3E 00-00 00-00 DE-00 00-00 00-00 00-00> (27 ms)
...
...
[  PASSED  ] 107 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] SpectralClusteringTests/SpectralClusteringTestF.Result/5, where GetParam() = 40-byte object <88-13 00-00 64-00 00-00 08-00 00-00 08-00 00-00 28-00 00-00 05-00 00-00 9A-99 99-3E 00-00 00-00 DE-00 00-00 00-00 00-00>

 1 FAILED TEST
CMake Error at run_gpu_test.cmake:26 (execute_process):
  execute_process failed command indexes:

I'll re-kick the tests on #1453.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by rerunning SpectralClusteringTests/SpectralClusteringTestF.Result/5 and comparing it with the failed CI run from #1453. Then inspect the eigensolver failure reported at eig.cuh:138 and the spectral clustering test setup. Done means the intermittent failure is reproduced and resolved or its cause is clearly documented with a reliable test outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.