lance-format / lance-format/lance
bug: a panic in the shuffle transform aborts the build instead of returning an error
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
IvfIndexBuilder::shuffle_data and reshuffle_partitions in rust/lance/src/index/vector/builder.rs run the transform for each batch in a spawned task and then unwrap the join result:
.buffered(get_num_compute_intensive_cpus())
.map(|x| x.unwrap())
A JoinError means the task panicked or was cancelled, so any panic inside the transform closure is re-raised in whatever task is draining the stream rather than returned to the caller of build().
There is a reachable panic in that closure. When a precomputed partitions file is in play, the closure attaches the partition ids with
batch = batch
.try_with_column(PART_ID_FIELD.clone(), Arc::new(part_ids.clone()))
.expect("failed to add part id column");
and try_with_column rejects a duplicate column name, so a batch that already carries __ivf_part_id panics here. The panic then travels through the unwrap above. That combination is what accelerator=... together with precomputed_partition_dataset=... used to produce.
Expected behavior
Attaching the partition ids should report a duplicate column as an invalid_input error naming the schema, and a panicking transform task should come back as an error from the shuffle rather than as a panic in the consumer.
Lance version
13.0.0-beta.4 (main)
Language binding
Rust
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance/src/index/vector/builder.rs with IvfIndexBuilder::shuffle_data and reshuffle_partitions, then trace the buffered stream's JoinError handling and the try_with_column call for PART_ID_FIELD. Reproduce the accelerator plus precomputed_partition_dataset case; done means duplicate partition columns produce an invalid_input error naming the schema and transform panics return an error from build() rather than aborting it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100