apache / apache/arrow-rs

Deprecate `values`/`keys`/etc in variable length arrays like list/map/binary in favor of 2 functions `actual_values` and `underlying_values` or something

Open
#7,992 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

Slicing on a variable length arrays like list/map/binary, does not slice the underlying values but instead only slice the offsets due to performance.

However when wanting to get the underlying values for a list/map/binary, calling `values` (on list for example) give you the child values even the ones that are not referred to by the current slice

I've lost count on the number of times I saw or had bugs with getting the values instead of the the actual values that the list point to.

this is an example where it's counter intuitive

```rust
let list = ListArray::from_iter_primitive::(vec![
Some(vec![Some(1), Some(2)]),
None,
Some(vec![Some(3), None, Some(5)]),
]);

let list = list.slice(1, 2);

// [null, [3, null, 5]]
println!("{:?}", list);

// [1, 2, 3, null, 5]
println!("{:?}", list.values());
```

we added comments on list `values` that mention that `The list array may not refer to all values in the `values` array ...` but this is still not enough IMO.

also, creating 2 functions that explicitly state what is returned will force the developer to think there is a cost (for example, casting list to string, you should only cast the sliced underlying values and not the entire underlying values and then slice)

-----

References to related changes/bugs:
- https://github.com/apache/arrow-rs/pull/7037 - fix for a bug I had in concat of sliced list
- https://github.com/apache/arrow-rs/issues/4409 - IPC slice
- https://github.com/apache/arrow-rs/issues/7993 - If the function would be explicit I believe it could have done or at least be aware of in the first place

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.