lance-format / lance-format/lance
perf: reading a shuffle partition plans one empty range per flush group it is absent from
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
TwoFileShuffleReader::partition_ranges in rust/lance-index/src/vector/v3/shuffler.rs builds one range per flush group, whether or not the partition has rows in that group:
for batch_idx in 0..self.num_batches {
let end_index = batch_idx * self.num_partitions + partition_id;
let start = if end_index == 0 { 0 } else { offsets[end_index - 1] };
ranges.push(start..offsets[end_index]);
}
A group the partition is absent from gives start == end, and those zero-length ranges go straight into ReadBatchParams::Ranges for the data file read. The on-demand offsets path does the same thing a few lines below.
The absent case is the common one, not the exception: a build with many partitions and many flush groups has each partition present in a small fraction of the groups. With 4096 partitions and 100 groups, reading one partition plans 100 ranges where a handful carry rows.
Expected behavior
Plan ranges only for the groups that hold rows, on both offset paths.
Lance version
13.0.0-beta.4 (main)
Language binding
Rust
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read rust/lance-index/src/vector/v3/shuffler.rs, starting at TwoFileShuffleReader::partition_ranges and the on-demand offsets path below it. Trace how offsets become ReadBatchParams::Ranges, then check the existing shuffle reader tests or test entry points for coverage. Done means both paths plan ranges only for flush groups containing rows, without changing reads for present groups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100