lance-format / lance-format/lance
Don't cache rep/def information by default in ComplexAllNullScheduler
Open
@Xuanwo is already working on this.
Since Jan 14, 2026.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
In ComplexAllNullScheduler we load the entire rep/def information for the page into memory during the initialize page and store it as part of the search cache. This is potentially expensive for RAM (especially since we do not have compression, see https://github.com/lancedb/lance/issues/4885). Once #4885 is solved we should only have a single buffer which means we should treat this as all other column types, not cache by default, and use a single IOP to fetch a random value.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.