lance-format / lance-format/lance
Don't require the search cache for full scans
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
If we are doing a full scan of a disk page we should not need to search cache. This means we can avoid IOPS (e.g. to load the miniblock repetition index) and also reduce compute work. We should investigate what optimizations are meaningful and apply them.
- Mini block layout shouldn't load the metadata at initialization time and instead load it at read time
- Mini block layout should never load the repetition index
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the full-scan path and the Mini block layout initialization and read paths. Check which metadata and repetition-index loads occur during full scans, then verify that avoiding them preserves scan behavior while reducing cache I/O and compute work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100