apache / apache/arrow-rs

Casting a sliced list to FixedSizeList silently returns the wrong values

Open
#10,975 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

### Describe the bug

Casting a sliced `ListArray` (or `LargeListArray`) to a `FixedSizeList` silently returns the wrong values: the result is taken from the start of the child array rather than from where the slice's offsets point. No error is raised — the data is just shifted.

`cast_list_to_fixed_size_list` reads the values as `array.values().slice(0, cap)` on the "all slices were the correct length" fast path, and starts `last_pos` at `0` on the padding path, neither of which accounts for `array.offsets()[0]` being non-zero after a slice.

https://github.com/apache/arrow-rs/blob/main/arrow-cast/src/cast/list.rs (in `cast_list_to_fixed_size_list`):

```rust
let values = match last_pos {
0 if !is_prev_empty => array.values().slice(0, cap), // All slices were the correct length
...
```

Still present on `main` as of today.

### To Reproduce

```rust
use std::sync::Arc;

use arrow::array::{ListArray, types::Int32Type};
use arrow::compute::cast;
use arrow::datatypes::{DataType, Field};

fn main() {
let list = ListArray::from_iter_primitive::(vec![
Some(vec![Some(1), Some(2)]),
Some(vec![Some(3), Some(4)]),
Some(vec![Some(5), Some(6)]),
]);

let to = DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Int32, true)), 2);

println!("unsliced: {:?}", cast(&list, &to).unwrap());

let sliced = list.slice(1, 2);
println!("sliced: {:?}", cast(&sliced, &to).unwrap());
}
```

The unsliced cast is correct. The sliced cast prints rows `[1, 2]` and `[3, 4]`, i.e. the first two rows of the original array rather than the two rows the slice covers.

### Expected behavior

The sliced cast should produce rows `[3, 4]` and `[5, 6]`, matching `cast(&list, &to).unwrap().slice(1, 2)`.

For comparison, `cast_list_values` (list → list of another item type) keeps the sliced offsets and casts the whole child array, so it handles a slice correctly. `cast_list_view_to_fixed_size_list` also looks correct, since it reads each element through `value_offset(idx)`.

### Additional context

Reproduced with `arrow` 58.4.0 and read on `main`. Slicing is easy to hit without doing it explicitly — a record batch sliced by the caller, or a list column coming out of a re-chunked stream — and because the cast succeeds, the wrong values are written or returned with nothing to indicate it. Found while writing a fixed-size-list column in LanceDB, where rows landed one position out of step.

I'm happy to open a PR if the fix — rebasing off `array.offsets()[0]` in both paths — looks right to you.

Contributor guide

Open the contributing guide

Research direction

Start in arrow-cast/src/cast/list.rs at cast_list_to_fixed_size_list, reviewing both the fast path and padding path against sliced offsets. Reproduce the issue with the provided ListArray example; done means the sliced cast returns [3, 4] and [5, 6], matching the sliced unsliced cast result.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.