apache / apache/arrow

[C++][Parquet] Improve parquet reading performance for String/Binary type based on Buffer operations instead of BinaryArrayBuilder

Open
#32,970 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Improve parquet reading performance for String/Binary type based on Buffer operations instead of BinaryArrayBuilder.

 

Currently, parquet String/Binary type reading performance is not good enough. For fixedwidth types, the Reader has adopted Buffer operations  instead of ArrayBuilder interfaces.

 

So it‘s better to use Buffer operations in String/Binary type reading.

 

**Reporter**: [xiangxiang Shen](https://issues.apache.org/jira/browse/ARROW-17735) / @zhixingheyi-tian
**Assignee**: [xiangxiang Shen](https://issues.apache.org/jira/browse/ARROW-17735) / @zhixingheyi-tian
#### PRs and other links:
- [GitHub Pull Request #14353](https://github.com/apache/arrow/pull/14353)

**Note**: *This issue was originally created as [ARROW-17735](https://issues.apache.org/jira/browse/ARROW-17735). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Parquet String/Binary reading path and its current BinaryArrayBuilder use, then compare it with the existing fixed-width Buffer operations. The work is complete when String/Binary reads use the intended Buffer-based approach and the reported reading-performance problem is addressed; pull request #14353 is already linked in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.