lance-format / lance-format/lance
Better default IVF_PQ parameters
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
One user found we get comparatively bad recall performance out-of-the-box compared to other systems [^1]:
[^1]: https://github.com/lancedb/lancedb/discussions/2382 and https://github.com/user-attachments/files/20122347/VectorDBBench.UI.pdf
I think we could likely do a lot better out of the box. Right now, we expect users to figure this out themselves, but the parameters aren't straight forward to optimize. What they have are the dimension of vectors and size of dataset, and a desired recall. They want to figure out how to get that recall and maximize performance. There are several knobs to do this, but they have indirect effects and all interact with one another. The system basically looks like this:
flowchart TD
A(["Dimension"]) --> B["num_sub_vectors"]
C(["Num rows"]) --> D["num_partitions"]
A --> E["num_bits"]
B --> I["Compression"] --> H["Latency"]
D --> J["Pruning"] --> F["Recall"]
G["n_probes"] --> J
E --> I
I --> F
J --> H
I think given the inputs dimension, num_rows, and desired recall, we should be able to give decent defaults for num_sub_vectors, num_bits, num_partitions and n_probes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name a file, test, or entry point. Start by tracing the IVF_PQ default parameter handling and its recall and performance evaluation; the work is done when dimension, row count, and target recall produce validated defaults for all four parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100