lance-format / lance-format/lance
perf: U64Segment::slice decodes and re-encodes a Range instead of slicing it in O(1)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
U64Segment::slice decodes the segment's values into a Vec<u64> and re-encodes them through from_slice, for every variant. For Range that work is redundant: a contiguous subslice of a range is the same range, so the answer is always Range(start + offset .. start + offset + len).
rechunk_sequences slices a segment once per output chunk, and a freshly written fragment's row-id sequence is exactly one Range segment (RowIdSequence::from(next_row_id..next_row_id + physical_rows)). So every stable-row-id compaction, update and merge insert decodes and re-encodes a whole sequence per chunk, which is O(rows) work plus an 8-byte-per-row intermediate Vec where O(1) would do. Compaction repeats the same pass for the two row-version columns through rechunk_version_sequences.
Sequences whose fragments carry deletions are masked to a gapped encoding first, so those keep the generic path either way.
Steps to reproduce
use lance_table::rowids::{rechunk_sequences, RowIdSequence};
// One Range segment, 10M row ids, split into 10 fragments of 1M.
let sequence = RowIdSequence::from(0..10_000_000u64);
let chunks = rechunk_sequences(vec![sequence], vec![1_000_000u64; 10], false).unwrap();
assert_eq!(chunks.len(), 10);
Each of the 10 slices decodes 1M u64 values into a Vec and re-encodes them. Measured with --profile release-no-lto (same binary, only the fast path toggled): about 38 ms per rechunk today against about 1.6 us with an O(1) slice.
Expected behavior
Slicing a Range segment is O(1) and allocation-free.
Lance version
13.0.0-beta.3 (main)
Language binding
Rust
Environment
macOS ARM, local storage. Not platform-specific.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with U64Segment::slice and trace how rechunk_sequences uses it for RowIdSequence ranges; also inspect rechunk_version_sequences for the related path. Use the reproduction with a 10M-row range as a baseline, then verify range slicing is allocation-free and retains the expected chunks and performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100