lance-format / lance-format/lance

perf: U64Segment::slice decodes and re-encodes a Range instead of slicing it in O(1)

Open Beginner friendly
#9,308 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

U64Segment::slice decodes the segment's values into a Vec<u64> and re-encodes them through from_slice, for every variant. For Range that work is redundant: a contiguous subslice of a range is the same range, so the answer is always Range(start + offset .. start + offset + len).

rechunk_sequences slices a segment once per output chunk, and a freshly written fragment's row-id sequence is exactly one Range segment (RowIdSequence::from(next_row_id..next_row_id + physical_rows)). So every stable-row-id compaction, update and merge insert decodes and re-encodes a whole sequence per chunk, which is O(rows) work plus an 8-byte-per-row intermediate Vec where O(1) would do. Compaction repeats the same pass for the two row-version columns through rechunk_version_sequences.

Sequences whose fragments carry deletions are masked to a gapped encoding first, so those keep the generic path either way.

Steps to reproduce
use lance_table::rowids::{rechunk_sequences, RowIdSequence};

// One Range segment, 10M row ids, split into 10 fragments of 1M.
let sequence = RowIdSequence::from(0..10_000_000u64);
let chunks = rechunk_sequences(vec![sequence], vec![1_000_000u64; 10], false).unwrap();
assert_eq!(chunks.len(), 10);

Each of the 10 slices decodes 1M u64 values into a Vec and re-encodes them. Measured with --profile release-no-lto (same binary, only the fast path toggled): about 38 ms per rechunk today against about 1.6 us with an O(1) slice.

Expected behavior

Slicing a Range segment is O(1) and allocation-free.

Lance version

13.0.0-beta.3 (main)

Language binding

Rust

Environment

macOS ARM, local storage. Not platform-specific.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with U64Segment::slice and trace how rechunk_sequences uses it for RowIdSequence ranges; also inspect rechunk_version_sequences for the related path. Use the reproduction with a 10M-row range as a baseline, then verify range slicing is allocation-free and retains the expected chunks and performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.