lance-format / lance-format/lance
Implement vector clustering based on BFS HNSW order
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We would like to increase the chance that similar vectors are nearby in data files. One idea is to order first by IVF partition, and then again using depth-first-search order of HNSW index.
We should do benchmarking to validate this meaningfully improves query performance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing IVF partitioning and HNSW index paths, then identify how data-file ordering is currently produced. Compare the proposed IVF-then-HNSW traversal ordering with the current layout and define benchmarks that measure query performance. Done means a validated clustering approach and benchmark results showing whether it improves queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100