[Python][FlightRPC] When using FlightDataStream yielding Tables means that IPCWriteOptions is not respected.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hi there!
I've been working with Arrow Flight in Python and came across something that might need a bit of attention. When using a `pa.flight.GeneratorStream`, if the generator yields a `Table` or a `RecordBatchReader`, it looks like the code in `_flight.pyx` (line 2028) doesn't pass the `IPCWriteOptions` set for the `GeneratorStream` to the newly created `RecordBatchStream`. As a result, if you want to compress the output stream, it doesn't get compressed.
https://github.com/apache/arrow/blob/f078942ce2df68de8f48c3b4233132133601ca53/python/pyarrow/_flight.pyx#L2038-L2042
I would have sent a PR for this, but it seems like the Python reference to the `IPCWriteOptions` isn't stored on the `GeneratorStream`. To address this, either a new constructor for `RecordBatchStream` could be created to allow the C-based `IPCWriteOptions` to be passed along, or a Python reference to the options needs to be retained.
### Component(s)
FlightRPC, Python
Contributor guide
Assessment
This issue has not been assessed yet.