apache / apache/arrow-rs

Refactor list -> fixedsizelist cast to use `take` instead of `MutableArraydata`

Open
#9,339 1 comment 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

I've seen some PRs where replacing usage of `MutableArrayData` with the `take` kernel can significantly improve performance

- Example from DataFusion: https://github.com/apache/datafusion/pull/20049

Consider following this approach for the code that casts list/listviews to fixedsizelists:

https://github.com/apache/arrow-rs/blob/dc99a52498ec25dae1fdc12618e992764486b88c/arrow-cast/src/cast/list.rs#L143-L154

https://github.com/apache/arrow-rs/blob/dc99a52498ec25dae1fdc12618e992764486b88c/arrow-cast/src/cast/list.rs#L227-L233

**Describe the solution you'd like**

Use `take` kernel to construct values array instead of `MutableArrayData`. Would need to see some benchmarks to prove this is a performance benefit. We also need to be careful of edge cases, see my comment:

- https://github.com/apache/arrow-rs/pull/9274#discussion_r2749221079

**Describe alternatives you've considered**

If this isn't a significant performance benefit we leave the implementation as is.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with arrow-cast/src/cast/list.rs at the two referenced ranges, then review the edge case in the arrow-rs#9274 discussion. Compare the existing MutableArrayData path with the take kernel and run suitable benchmarks. Done means list/listview-to-fixedsizelist casting preserves edge-case behavior and shows a meaningful performance benefit; otherwise retain the current implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.