Refactor list -> fixedsizelist cast to use `take` instead of `MutableArraydata`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I've seen some PRs where replacing usage of `MutableArrayData` with the `take` kernel can significantly improve performance
- Example from DataFusion: https://github.com/apache/datafusion/pull/20049
Consider following this approach for the code that casts list/listviews to fixedsizelists:
https://github.com/apache/arrow-rs/blob/dc99a52498ec25dae1fdc12618e992764486b88c/arrow-cast/src/cast/list.rs#L143-L154
https://github.com/apache/arrow-rs/blob/dc99a52498ec25dae1fdc12618e992764486b88c/arrow-cast/src/cast/list.rs#L227-L233
**Describe the solution you'd like**
Use `take` kernel to construct values array instead of `MutableArrayData`. Would need to see some benchmarks to prove this is a performance benefit. We also need to be careful of edge cases, see my comment:
- https://github.com/apache/arrow-rs/pull/9274#discussion_r2749221079
**Describe alternatives you've considered**
If this isn't a significant performance benefit we leave the implementation as is.
**Additional context**
Contributor guide
Research direction
Start with arrow-cast/src/cast/list.rs at the two referenced ranges, then review the edge case in the arrow-rs#9274 discussion. Compare the existing MutableArrayData path with the take kernel and run suitable benchmarks. Done means list/listview-to-fixedsizelist casting preserves edge-case behavior and shows a meaningful performance benefit; otherwise retain the current implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100