NVIDIA / NVIDIA/cuvs

Multi-GPU out of core KMeans progress tracking issue

Open
#2,296 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

This issue aims at tracking progress on the multi-GPU out of core KMeans implementation and benchmarking.

Please tell me if anything was forgotten. Also, if you add a new issue/PR, please list it here and link this issue as its parent so we can keep track of everything that is going on.

Main Tasks

  • Multi-GPU batched KMeans core (#2017)
  • Multi-GPU KMeans implementations merge + cleanup (#2015)
  • SNMG Batched KMeans Python API (#2154)
  • Handle multiple partitions per rank/worker + API merges (#2066)
  • Handle multiple partitions per rank/worker (cuML side) (rapidsai/cuml#8201 and rapidsai/cuml#8084)
  • Evaluate KMeansPlusPlus Init For Out of Core (#2224)
  • SNMG Batched KMeans Python API benchmark (#2149)
  • Large scale multi-node Dask KMeans benchmark (rapidsai/cuml#8198)

Improvements and bugfixes

  • Out-of-core K-means improvements (#2292)
  • Apply ABI Breaking Changes for KMeans (#2147)
  • Reuse Precomputed Norms for Inertia Computation (#2057)
  • Evaluate Prefetch for Batched Kmeans (#1917)
  • Clamp max_iter in OOC KMeans test (#2256)
  • KMeans ignores sample_weight when computing inertia and score (rapidsai/cuml#8530)
  • KMeans.transform returns squared distances instead of Euclidean distances (rapidsai/cuml#8536)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the unchecked tasks in this tracking issue, starting with the linked issues #2224, #2149, #8198, and #2292. Use the listed cuML issues and related benchmarks to understand the remaining multi-GPU out-of-core KMeans work. Done means completing a scoped task and updating this checklist with the relevant issue or pull request link.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.