apache / apache/arrow

[Python][FlightRPC] When using FlightDataStream yielding Tables means that IPCWriteOptions is not respected.

Open
#43,779 0 comments 0 reactions 0 assignees View on GitHub
Component: FlightRPC Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hi there!

I've been working with Arrow Flight in Python and came across something that might need a bit of attention. When using a `pa.flight.GeneratorStream`, if the generator yields a `Table` or a `RecordBatchReader`, it looks like the code in `_flight.pyx` (line 2028) doesn't pass the `IPCWriteOptions` set for the `GeneratorStream` to the newly created `RecordBatchStream`. As a result, if you want to compress the output stream, it doesn't get compressed.

https://github.com/apache/arrow/blob/f078942ce2df68de8f48c3b4233132133601ca53/python/pyarrow/_flight.pyx#L2038-L2042

I would have sent a PR for this, but it seems like the Python reference to the `IPCWriteOptions` isn't stored on the `GeneratorStream`. To address this, either a new constructor for `RecordBatchStream` could be created to allow the C-based `IPCWriteOptions` to be passed along, or a Python reference to the options needs to be retained.

### Component(s)

FlightRPC, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.