Casting a sliced list to FixedSizeList silently returns the wrong values
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
### Describe the bug
Casting a sliced `ListArray` (or `LargeListArray`) to a `FixedSizeList` silently returns the wrong values: the result is taken from the start of the child array rather than from where the slice's offsets point. No error is raised — the data is just shifted.
`cast_list_to_fixed_size_list` reads the values as `array.values().slice(0, cap)` on the "all slices were the correct length" fast path, and starts `last_pos` at `0` on the padding path, neither of which accounts for `array.offsets()[0]` being non-zero after a slice.
https://github.com/apache/arrow-rs/blob/main/arrow-cast/src/cast/list.rs (in `cast_list_to_fixed_size_list`):
```rust
let values = match last_pos {
0 if !is_prev_empty => array.values().slice(0, cap), // All slices were the correct length
...
```
Still present on `main` as of today.
### To Reproduce
```rust
use std::sync::Arc;
use arrow::array::{ListArray, types::Int32Type};
use arrow::compute::cast;
use arrow::datatypes::{DataType, Field};
fn main() {
let list = ListArray::from_iter_primitive::(vec![
Some(vec![Some(1), Some(2)]),
Some(vec![Some(3), Some(4)]),
Some(vec![Some(5), Some(6)]),
]);
let to = DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Int32, true)), 2);
println!("unsliced: {:?}", cast(&list, &to).unwrap());
let sliced = list.slice(1, 2);
println!("sliced: {:?}", cast(&sliced, &to).unwrap());
}
```
The unsliced cast is correct. The sliced cast prints rows `[1, 2]` and `[3, 4]`, i.e. the first two rows of the original array rather than the two rows the slice covers.
### Expected behavior
The sliced cast should produce rows `[3, 4]` and `[5, 6]`, matching `cast(&list, &to).unwrap().slice(1, 2)`.
For comparison, `cast_list_values` (list → list of another item type) keeps the sliced offsets and casts the whole child array, so it handles a slice correctly. `cast_list_view_to_fixed_size_list` also looks correct, since it reads each element through `value_offset(idx)`.
### Additional context
Reproduced with `arrow` 58.4.0 and read on `main`. Slicing is easy to hit without doing it explicitly — a record batch sliced by the caller, or a list column coming out of a re-chunked stream — and because the cast succeeds, the wrong values are written or returned with nothing to indicate it. Found while writing a fixed-size-list column in LanceDB, where rows landed one position out of step.
I'm happy to open a PR if the fix — rebasing off `array.offsets()[0]` in both paths — looks right to you.
Contributor guide
Research direction
Start in arrow-cast/src/cast/list.rs at cast_list_to_fixed_size_list, reviewing both the fast path and padding path against sliced offsets. Reproduce the issue with the provided ListArray example; done means the sliced cast returns [3, 4] and [5, 6], matching the sliced unsliced cast result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100