Lucene: Unset params in CAGRA w/ build_algo = IVF-PQ not using heuristics
@imotov is already working on this.
Since Aug 17, 2026.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Problem
CAGRA can tune IVF-PQ parameters (n_lists, pq_dim, n_probes, etc.) from dataset size and dimension — see ivf_pq_params and the Java copy in getCuVSIvfPqParams().
If a user sets only a few knobs (e.g. n_lists and pq_bits in a sweep), they expect the rest to be filled by those heuristics. That works with Strategy.HEURISTIC (≥ 5M vectors). It does not work with Strategy.CUSTOM, which is what you get when explicitly configuring IVF_PQ — unset fields fall back to fixed Builder defaults like n_lists=1024 and kmeans_trainset_fraction=0.5, regardless of dataset size.
Native code already does the right thing — heuristic baseline, then overlay only supplied fields (cagra.cpp) — but Java passes every field, clobbering the baseline. Only pq_dim=0 gets a partial escape hatch via calculate_pq_dim.
Expected
explicit value > dataset heuristic > static default
Per parameter. Match native behavior: baseline from ivf_pq_params(dataset), overlay only what the caller set.
Example
10M × 1536d — user sets only n_lists=5000, pq_bits=4:
| Param | Should be | Today |
|---|---|---|
kmeans_n_iters |
10 | 20 |
kmeans_trainset_fraction |
~0.1 | 0.5 |
n_probes |
~8 | 20 |
| search dtypes | fp16 | fp32 |
(pq_dim accidentally works via 0 → calculate_pq_dim; other fields don't.)
Fix
- Track explicitly set IVF-PQ keys.
- Baseline =
getCuVSIvfPqParams(N, D); overlay explicit keys only.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.