apache / apache/arrow

[C++][Python] Add conversion from RecordBatchFileReader to RecordBatchReader

Open
#31,394 10 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

The suggested improvement is to introduce a conversion/adapter so that all batches from RecordBatchFileReader can be read one-by-one using RecordBatchReader.

Perhaps a new instance method RecordBatchFileReader.to_reader()? This would follow the suit of for instance the pyarrow.flight.MetadataRecordBatchReader which also has to_reader().

**Motivation**

Record Batches serialized into IPC file format can be read using RecordBatchFileReader. The interface of this reader is incompatible with RecordBatchReader.

This impacts for instance the Flight RPC DoGet, where it is not possible to efficiently (e.g. fully in C++) send out all data by using pyarrow.flight.RecordBatchStream. However, there may be other use cases where client code wants to read data batch-by-batch transparently, without caring about the serialization format.

Further background is here:

 

**Reporter**: [Lubo Slivka](https://issues.apache.org/jira/browse/ARROW-15969)

**Note**: *This issue was originally created as [ARROW-15969](https://issues.apache.org/jira/browse/ARROW-15969). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reading the RecordBatchFileReader and RecordBatchReader entry points, then compare the existing pyarrow.flight.MetadataRecordBatchReader.to_reader() behavior. Check how Flight RPC DoGet and pyarrow.flight.RecordBatchStream consume batches. Done means all batches from a file reader can be read one-by-one through the RecordBatchReader interface without depending on the serialization format.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.