apache / apache/arrow

[C++][Parquet] Add an asynchronous version of ReadRowGroup/ReadRowGroups

Open
#36,778 1 comment 1 reaction 1 assignee Claimed by @westonpace View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

We do actually already have an asynchronous version with the method `GetRecordBatchGenerator`. However, that method does not respect the batch size property and it is not possible to read less than 1 entire row group at a time. This makes it difficult to read large parquet files and makes the scanner's memory usage very dependent on a parquet file's row group size.

This PR is requesting a new method which is a closer analogue to the existing ReadRowGroup/ReadRowGroups methods. Once the scanner moves over to this new method then I think we can deprecate `GetRecordBatchGenerator`. I hesitate to replace it immediately as I don't want to introduce any breaking changes into the existing scanner path.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.