lance-format / lance-format/lance

bug: rechunk_sequences rejects a valid input when a sequence ends with an empty segment

Open Beginner friendly
#9,301 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

With allow_incomplete = false, rechunk_sequences errors when segments are left over after the last chunk is filled. Its fill loop drains empty segments only while a chunk still needs rows, so a trailing empty segment is still sitting in the iterator when the final "too many segments" check runs, and a call whose chunk sizes sum to exactly the number of row ids fails.

Empty segments are tolerated in every other position: test_row_id_sequence_rechunk_with_empty_segments covers one inside a chunk, one at a chunk boundary, and several in a row. The trailing position is the gap.

I could not find a path that reaches this from a public API today. The four callers (update, merge insert in two places, compaction) all pass sequences that cannot end with an empty segment: production row removal goes through RowIdSequence::mask, which drops empty segments, and the captured sequences come from streams whose scan nodes filter zero-row batches out. The capture path itself has no num_rows() > 0 guard (extract_row_ids captures every batch it polls, and RowIdSequence::from(&[]) yields one empty segment), so that invariant lives in the upstream nodes, not in the capture code.

Steps to reproduce
use lance_table::rowids::{rechunk_sequences, RowIdSequence};

// two row ids, one chunk of two, plus a trailing segment that carries nothing
let sequences = vec![RowIdSequence::from(0..2), RowIdSequence::from(10..10)];
let err = rechunk_sequences(sequences, vec![2], false).unwrap_err();
// InvalidInput: Got too many segments for the provided chunk lengths.
// Processed 1 chunks out of 1 expected
Expected behavior

Ok with one chunk holding [0, 1]. An empty segment carries no row ids, so it is not excess.

Lance version

13.0.0-beta.3 (main)

Language binding

Rust

Environment

macOS ARM, local storage. Not platform-specific.

Logs / traceback
InvalidInput { source: "Got too many segments for the provided chunk lengths. Processed 1 chunks out of 1 expected", location: rust/lance-table/src/rowids.rs:837 }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance-table/src/rowids.rs near the rechunk_sequences check at line 837, and run the supplied reproduction plus test_row_id_sequence_rechunk_with_empty_segments. Confirm that a trailing empty RowIdSequence is ignored when chunk lengths are fully satisfied and the example returns one chunk containing [0, 1].

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.