lance-format / lance-format/lance
Potential scanner optimization for late materialization plus recheck
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Adding this as a follow-up as it seems a pretty niche optimization that may require a fair amount of work.
Imagine we have a query contains(caption, 'elephant') AND my_col = 75 where we have an ngram index on caption and no index on my_col.
Now let's assume there are 100M rows and 10M rows match the first pass of the ngram filter.
Let's also assume there are 1K rows that satisfy my_col = 75.
Today, the way that filtered read works, since caption is indexed we will use it in the early materialization phase. This means we load all 10M potential matches for caption.
We could speed up this query with late materialization.
Note: if my_col has a scalar index then this optimization is not needed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the scanner and filtered-read implementation responsible for early materialization; the issue does not name specific files or tests. Compare the current behavior with the described ngram-only scenario, then define and validate a late-materialization path whose results match the existing query semantics without requiring a scalar index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100