[C++][IPC] Allow a more fine-grained column selection for reading data with nested fields
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
### Missing feature: fine-grained column selection/projection
Currently (Apache Arrow 13.0.0), when reading data from IPC via, e.g., `RecordBatchStreamReader::Open`, a list of top-level fields can be selected via the `IpcReadOptions::included_fields` member.
This is useful, however, in presence of nested fields, sometimes it's necessary to select only some leaf nodes of the nested columns.
### Suggested Improvement
It would be nice to provide an additional way to select individual nested fields.
### Implementation strategy
There are several ways how this could be achieved (assuming that changing the semantics of `IpcReadOptions::included_fields` is not an option):
* extending the `IpcReadOptions` type to include a `vector included_flattened_fields` member (indicating the index in the "flattened" schema), similar to how it is done for Parquet (see below)
* similarly, extending the `IpcReadOptions` type to include a `vector> included_field_paths` member
* similarly, extending the `IpcReadOptions` type to include a `vector included_field_refs` member
### Similar approaches in the code base
If I see this correctly, in the Apache Parquet implementation, this achieved by specifying the list of flattened field indices (e.g. in [`GetFieldReaders`](https://github.com/apache/arrow/blob/apache-arrow-13.0.0/cpp/src/parquet/arrow/reader.cc#L234), which is called in `FileReader::GetRecordBatchReader`).
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.