Multi-GPU out of core KMeans progress tracking issue
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
This issue aims at tracking progress on the multi-GPU out of core KMeans implementation and benchmarking.
Please tell me if anything was forgotten. Also, if you add a new issue/PR, please list it here and link this issue as its parent so we can keep track of everything that is going on.
Main Tasks
- Multi-GPU batched KMeans core (#2017)
- Multi-GPU KMeans implementations merge + cleanup (#2015)
- SNMG Batched KMeans Python API (#2154)
- Handle multiple partitions per rank/worker + API merges (#2066)
- Handle multiple partitions per rank/worker (cuML side) (rapidsai/cuml#8201 and rapidsai/cuml#8084)
- Evaluate KMeansPlusPlus Init For Out of Core (#2224)
- SNMG Batched KMeans Python API benchmark (#2149)
- Large scale multi-node Dask KMeans benchmark (rapidsai/cuml#8198)
Improvements and bugfixes
- Out-of-core K-means improvements (#2292)
- Apply ABI Breaking Changes for KMeans (#2147)
- Reuse Precomputed Norms for Inertia Computation (#2057)
- Evaluate Prefetch for Batched Kmeans (#1917)
- Clamp max_iter in OOC KMeans test (#2256)
- KMeans ignores sample_weight when computing inertia and score (rapidsai/cuml#8530)
- KMeans.transform returns squared distances instead of Euclidean distances (rapidsai/cuml#8536)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the unchecked tasks in this tracking issue, starting with the linked issues #2224, #2149, #8198, and #2292. Use the listed cuML issues and related benchmarks to understand the remaining multi-GPU out-of-core KMeans work. Done means completing a scoped task and updating this checklist with the relevant issue or pull request link.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100