[Python] Cast one chunk at a time in ChunkedArray ArrowStream export with requested type
Open
Component: Python
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Follow-up on https://github.com/apache/arrow/pull/40818, see https://github.com/apache/arrow/pull/40818#discussion_r1551271374 for context.
`ChunkedArray.__arrow_c_stream__` currently casts the full ChunkedArray up front when providing a `requested_type`. Instead, we can also catst one chunk at a time when the stream is being read, to reduce memory consumption.
This is similar to what was implemented for RecordBatchReader in https://github.com/apache/arrow/pull/40070 (`CastingRecordBatchReader`)
Contributor guide
Assessment
This issue has not been assessed yet.