apache / apache/arrow

[C++][Parquet] support passing a RowRange to RecordBatchReader

Open
#38,865 8 comments 0 reactions 1 assignee Claimed by @binmahone View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Currently GetRecordBatchReader API accepts row_group_indices and column_indices. It would be nice to extend the API to accept one more parameter: A row_ranges indicating a subset of rows to be retrieved. With the provided row_ranges, RecordBatchReader can skip unnecessary pages (by comparing the row_ranges with the might-exist page index) as well as unwanted rows.

- original:

```c++
::arrow::Status GetRecordBatchReader(const std::vector& row_group_indices,
const std::vector& column_indices,
std::shared_ptr<::arrow::RecordBatchReader>* out);
```

- proposal:

```c++
::arrow::Status GetRecordBatchReader(
const std::vector& row_group_indices, const std::vector& column_indices,
const std::shared_ptr>& row_ranges_map, # a row_ranges per Row Group
std::shared_ptr<::arrow::RecordBatchReader>* out);
```
API clients can query page index or other kinds of index (e.g. external secondary index) to construct the row_ranges.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.