lance-format / lance-format/lance
Unexpectedly slow scalar index performance
Open
@westonpace is already working on this.
Since Jan 30, 2024.
A-index
performance
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
This was reported from Discord. When querying an indexed column with a very simple query (unique_id_col = some_id) there was a lot of CPU time spent in the materialize index node. It looks like we are always materializing the entire list of row IDs (this table had millions of rows) even if the allow list is very small. We can probably pick the smaller of the two lists to iterate on if there is no block list.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.