lance-format / lance-format/lance
bug: rechunk_sequences rejects a valid input when a sequence ends with an empty segment
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
With allow_incomplete = false, rechunk_sequences errors when segments are left over after the last chunk is filled. Its fill loop drains empty segments only while a chunk still needs rows, so a trailing empty segment is still sitting in the iterator when the final "too many segments" check runs, and a call whose chunk sizes sum to exactly the number of row ids fails.
Empty segments are tolerated in every other position: test_row_id_sequence_rechunk_with_empty_segments covers one inside a chunk, one at a chunk boundary, and several in a row. The trailing position is the gap.
I could not find a path that reaches this from a public API today. The four callers (update, merge insert in two places, compaction) all pass sequences that cannot end with an empty segment: production row removal goes through RowIdSequence::mask, which drops empty segments, and the captured sequences come from streams whose scan nodes filter zero-row batches out. The capture path itself has no num_rows() > 0 guard (extract_row_ids captures every batch it polls, and RowIdSequence::from(&[]) yields one empty segment), so that invariant lives in the upstream nodes, not in the capture code.
Steps to reproduce
use lance_table::rowids::{rechunk_sequences, RowIdSequence};
// two row ids, one chunk of two, plus a trailing segment that carries nothing
let sequences = vec![RowIdSequence::from(0..2), RowIdSequence::from(10..10)];
let err = rechunk_sequences(sequences, vec![2], false).unwrap_err();
// InvalidInput: Got too many segments for the provided chunk lengths.
// Processed 1 chunks out of 1 expected
Expected behavior
Ok with one chunk holding [0, 1]. An empty segment carries no row ids, so it is not excess.
Lance version
13.0.0-beta.3 (main)
Language binding
Rust
Environment
macOS ARM, local storage. Not platform-specific.
Logs / traceback
InvalidInput { source: "Got too many segments for the provided chunk lengths. Processed 1 chunks out of 1 expected", location: rust/lance-table/src/rowids.rs:837 }
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance-table/src/rowids.rs near the rechunk_sequences check at line 837, and run the supplied reproduction plus test_row_id_sequence_rechunk_with_empty_segments. Confirm that a trailing empty RowIdSequence is ignored when chunk lengths are fully satisfied and the example returns one chunk containing [0, 1].
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100