apache / apache/arrow

[C++][IPC] Allow a more fine-grained column selection for reading data with nested fields

Open
#37,358 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

### Missing feature: fine-grained column selection/projection
Currently (Apache Arrow 13.0.0), when reading data from IPC via, e.g., `RecordBatchStreamReader::Open`, a list of top-level fields can be selected via the `IpcReadOptions::included_fields` member.
This is useful, however, in presence of nested fields, sometimes it's necessary to select only some leaf nodes of the nested columns.

### Suggested Improvement

It would be nice to provide an additional way to select individual nested fields.

### Implementation strategy

There are several ways how this could be achieved (assuming that changing the semantics of `IpcReadOptions::included_fields` is not an option):
* extending the `IpcReadOptions` type to include a `vector included_flattened_fields` member (indicating the index in the "flattened" schema), similar to how it is done for Parquet (see below)
* similarly, extending the `IpcReadOptions` type to include a `vector> included_field_paths` member
* similarly, extending the `IpcReadOptions` type to include a `vector included_field_refs` member

### Similar approaches in the code base

If I see this correctly, in the Apache Parquet implementation, this achieved by specifying the list of flattened field indices (e.g. in [`GetFieldReaders`](https://github.com/apache/arrow/blob/apache-arrow-13.0.0/cpp/src/parquet/arrow/reader.cc#L234), which is called in `FileReader::GetRecordBatchReader`).

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.