Deprecate `values`/`keys`/etc in variable length arrays like list/map/binary in favor of 2 functions `actual_values` and `underlying_values` or something
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
Slicing on a variable length arrays like list/map/binary, does not slice the underlying values but instead only slice the offsets due to performance.
However when wanting to get the underlying values for a list/map/binary, calling `values` (on list for example) give you the child values even the ones that are not referred to by the current slice
I've lost count on the number of times I saw or had bugs with getting the values instead of the the actual values that the list point to.
this is an example where it's counter intuitive
```rust
let list = ListArray::from_iter_primitive::(vec![
Some(vec![Some(1), Some(2)]),
None,
Some(vec![Some(3), None, Some(5)]),
]);
let list = list.slice(1, 2);
// [null, [3, null, 5]]
println!("{:?}", list);
// [1, 2, 3, null, 5]
println!("{:?}", list.values());
```
we added comments on list `values` that mention that `The list array may not refer to all values in the `values` array ...` but this is still not enough IMO.
also, creating 2 functions that explicitly state what is returned will force the developer to think there is a cost (for example, casting list to string, you should only cast the sliced underlying values and not the entire underlying values and then slice)
-----
References to related changes/bugs:
- https://github.com/apache/arrow-rs/pull/7037 - fix for a bug I had in concat of sliced list
- https://github.com/apache/arrow-rs/issues/4409 - IPC slice
- https://github.com/apache/arrow-rs/issues/7993 - If the function would be explicit I believe it could have done or at least be aware of in the first place
Contributor guide
Assessment
This issue has not been assessed yet.