NVIDIA / NVIDIA/cuvs

[BUG] Multi-GPU KMeans convergence degradation with default random generator

Open
#1,829 5 comments 0 reactions 1 assignee View on GitHub

@vinaydes is already working on this.

Since Feb 23, 2026.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

with raft::random::GeneratorType::GenPhilox :

[mg::detail::fit] rank=1 iter=1 sqrdNormError=2.9123220703e+04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=1 sqrdNormError=2.9123222656e+04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=2 sqrdNormError=1.2270645833e-08 tol=1.0000000000e-04 inertia_check=0 done=1
[mg::detail::fit] rank=0 iter=2 sqrdNormError=1.2270649385e-08 tol=1.0000000000e-04 inertia_check=0 done=1

with raft::random::GeneratorType::GenPC (default) :

[mg::detail::fit] rank=1 iter=1 sqrdNormError=2.9187794922e+04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=1 sqrdNormError=2.9187775391e+04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=2 sqrdNormError=7.4286144227e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=2 sqrdNormError=7.4286144227e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=3 sqrdNormError=5.3626983427e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=3 sqrdNormError=5.3626988083e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=4 sqrdNormError=4.0814131498e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=4 sqrdNormError=4.0814126842e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=5 sqrdNormError=3.3664486837e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=5 sqrdNormError=3.3664491493e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=6 sqrdNormError=2.5890916586e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=6 sqrdNormError=2.5890930556e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=7 sqrdNormError=1.9509658450e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=7 sqrdNormError=1.9509647973e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=8 sqrdNormError=1.5524497721e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=8 sqrdNormError=1.5524502378e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=9 sqrdNormError=1.3301083818e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=9 sqrdNormError=1.3301081490e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=10 sqrdNormError=1.0619051754e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=10 sqrdNormError=1.0619052919e-03 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=11 sqrdNormError=9.2023389880e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=11 sqrdNormError=9.2023389880e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=12 sqrdNormError=8.3746155724e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=12 sqrdNormError=8.3746074233e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=13 sqrdNormError=8.0149032874e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=13 sqrdNormError=8.0149050336e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=14 sqrdNormError=6.5786024788e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=14 sqrdNormError=6.5785838524e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=15 sqrdNormError=5.4237182485e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=15 sqrdNormError=5.4237066070e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=16 sqrdNormError=4.5198111911e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=16 sqrdNormError=4.5198114822e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=17 sqrdNormError=4.0510416147e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=17 sqrdNormError=4.0510357940e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=18 sqrdNormError=3.8176175440e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=18 sqrdNormError=3.8176230737e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=19 sqrdNormError=3.7566581159e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=19 sqrdNormError=3.7566569517e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=0 iter=20 sqrdNormError=3.5087612923e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[mg::detail::fit] rank=1 iter=20 sqrdNormError=3.5087575088e-04 tol=1.0000000000e-04 inertia_check=0 done=0
[...]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.