lance-format / lance-format/lance

perf: reading a shuffle partition plans one empty range per flush group it is absent from

Open
#9,412 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

TwoFileShuffleReader::partition_ranges in rust/lance-index/src/vector/v3/shuffler.rs builds one range per flush group, whether or not the partition has rows in that group:

for batch_idx in 0..self.num_batches {
    let end_index = batch_idx * self.num_partitions + partition_id;
    let start = if end_index == 0 { 0 } else { offsets[end_index - 1] };
    ranges.push(start..offsets[end_index]);
}

A group the partition is absent from gives start == end, and those zero-length ranges go straight into ReadBatchParams::Ranges for the data file read. The on-demand offsets path does the same thing a few lines below.

The absent case is the common one, not the exception: a build with many partitions and many flush groups has each partition present in a small fraction of the groups. With 4096 partitions and 100 groups, reading one partition plans 100 ranges where a handful carry rows.

Expected behavior

Plan ranges only for the groups that hold rows, on both offset paths.

Lance version

13.0.0-beta.4 (main)

Language binding

Rust

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read rust/lance-index/src/vector/v3/shuffler.rs, starting at TwoFileShuffleReader::partition_ranges and the on-demand offsets path below it. Trace how offsets become ReadBatchParams::Ranges, then check the existing shuffle reader tests or test entry points for coverage. Done means both paths plan ranges only for flush groups containing rows, without changing reads for present groups.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.