lance-format / lance-format/lance

Potential scanner optimization for late materialization plus recheck

Open
#4,107 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Adding this as a follow-up as it seems a pretty niche optimization that may require a fair amount of work.

Imagine we have a query contains(caption, 'elephant') AND my_col = 75 where we have an ngram index on caption and no index on my_col.

Now let's assume there are 100M rows and 10M rows match the first pass of the ngram filter.
Let's also assume there are 1K rows that satisfy my_col = 75.

Today, the way that filtered read works, since caption is indexed we will use it in the early materialization phase. This means we load all 10M potential matches for caption.

We could speed up this query with late materialization.

Note: if my_col has a scalar index then this optimization is not needed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the scanner and filtered-read implementation responsible for early materialization; the issue does not name specific files or tests. Compare the current behavior with the described ngram-only scenario, then define and validate a late-materialization path whose results match the existing query semantics without requiring a scalar index.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.