[C++][Python] Add conversion from RecordBatchFileReader to RecordBatchReader
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
The suggested improvement is to introduce a conversion/adapter so that all batches from RecordBatchFileReader can be read one-by-one using RecordBatchReader.
Perhaps a new instance method RecordBatchFileReader.to_reader()? This would follow the suit of for instance the pyarrow.flight.MetadataRecordBatchReader which also has to_reader().
**Motivation**
Record Batches serialized into IPC file format can be read using RecordBatchFileReader. The interface of this reader is incompatible with RecordBatchReader.
This impacts for instance the Flight RPC DoGet, where it is not possible to efficiently (e.g. fully in C++) send out all data by using pyarrow.flight.RecordBatchStream. However, there may be other use cases where client code wants to read data batch-by-batch transparently, without caring about the serialization format.
Further background is here:
**Reporter**: [Lubo Slivka](https://issues.apache.org/jira/browse/ARROW-15969)
**Note**: *This issue was originally created as [ARROW-15969](https://issues.apache.org/jira/browse/ARROW-15969). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reading the RecordBatchFileReader and RecordBatchReader entry points, then compare the existing pyarrow.flight.MetadataRecordBatchReader.to_reader() behavior. Check how Flight RPC DoGet and pyarrow.flight.RecordBatchStream consume batches. Done means all batches from a file reader can be read one-by-one through the RecordBatchReader interface without depending on the serialization format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100