lance-format / lance-format/lance
GPU-enabled index training
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently the index training time is dominated by kmeans training, which can be sped up significantly via hardware acceleration.
Recently we added an API to take externally trained IVF centroids. So we could have an optional feature where the centroids are trained using pytorch or rapids then set the centroids explicitly.
Examples:
- https://pypi.org/project/kmeans-pytorch/
- https://medium.com/dropout-analytics/intro-to-k-means-clustering-with-cuml-b6d617e36456
https://github.com/lancedb/lance/blob/main/python/python/lance/dataset.py#L481
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the external IVF-centroid API and the dataset.py entry point linked at line 481, then review how kmeans training currently fits into index creation. Compare the referenced PyTorch and RAPIDS approaches and define an optional path that supplies externally trained centroids; done means GPU-assisted training can be selected without changing the existing default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100