lance-format / lance-format/lance
bug: filter_nan=False is silently ignored on CPU index builds
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
create_index(..., filter_nan=False) is documented as an unsafe speed knob: "False is UNSAFE, and will cause a crash if any null/nan values are present (and otherwise will not). Disables the null filter used for nullable columns. Obtains a small speed boost."
The parameter only reaches the torch helpers in lance/vector.py, and those calls sit inside the if accelerator is not None: block of _create_index_impl (one_pass_train_ivf_pq_on_accelerator(..., filter_nan=filter_nan) and one_pass_assign_ivf_pq_on_accelerator(..., filter_nan=filter_nan)). It is never placed in the kwargs dict that goes to Rust, and the CPU training path filters non-finite vectors unconditionally through filter_finite_training_data and the KeepFiniteVectors transform.
So on a CPU build, which is the default, the parameter decides nothing and nothing says so. A user who sets it is either expecting a speed boost they do not get, or believing they have taken on a crash risk they have not.
Expected behavior
Warn when filter_nan=False is passed without an accelerator, and document that the parameter applies to accelerator builds only.
Lance version
13.0.0-beta.4 (main)
Language binding
Python
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lance/vector.py at _create_index_impl and trace how filter_nan reaches the accelerator calls versus the Rust kwargs and CPU filtering path. Check the documentation for create_index and the existing filter_finite_training_data and KeepFiniteVectors behavior. Done means CPU use of filter_nan=False is clearly warned about and the documentation states that the parameter applies only to accelerator builds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100