apache / apache/arrow-rs

[Parquet] Remove uses of ArrayData in favor of Direct Array construction

Open
#9,128 2 comments 1 reaction 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- related to https://github.com/apache/arrow-rs/issues/9061

I noticed on https://github.com/apache/arrow-rs/issues/9061 that there is non trivial overhead to array. I am trying to improve make_array in parallel, but @tustvold had an even better idea in https://github.com/apache/arrow-rs/pull/9058#issuecomment-3712272488

> My 2 cents is it would be better to move the codepaths relying on ArrayData over to using the typed arrays directly, this should not only cut down on allocations but unnecessary validation and dispatch overheads.

An ArrayData has at least one extra allocation (for the Vec that holds Buffers) as well as a bunch of dynamic function calls. While this overhead is small individually, it is paid for every array so in aggregate it can be substantial

**Describe the solution you'd like**
Change relying on ArrayData over to using the typed arrays directly, this should not only cut down on allocations but unnecessary validation and dispatch overheads.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reading issues 9061 and pull request 9058, then search the Parquet codepaths for uses of ArrayData. Identify each affected construction path and its existing coverage before replacing the indirect paths with typed arrays. Done means the relevant Parquet paths no longer rely on ArrayData while preserving current behavior and reducing the described allocation and dispatch overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.