[Python][FlightRPC] IPC error using Python GeneratorStream for tables containing Categorical / DictionaryArray
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### pyarrow=13.0.0
Seeing the following error from the Flight client side when calling reader.read_all()
`
ArrowInvalid: IPC stream did not have the expected number (1) of dictionaries at the start of the stream
table.column_names
`
The issue is only observed when:
1. the server returns a "GeneratorStream" instead of "RecordBatchStream"
2. the table contains column of Categorical / DictionaryArray type
Server.py:
```
import pyarrow as pa
from pyarrow.flight import FlightServerBase, Location, ServerCallContext, Ticket, GeneratorStream
import pandas as pd
class FlightServer(FlightServerBase):
def __init__(self, host, port):
super().__init__(Location.for_grpc_tcp(host, port))
def do_get(self, context: ServerCallContext, ticket: Ticket):
df = pd.DataFrame.from_dict({
'col_1': pd.Categorical(['a', 'b', 'c', 'a', 'b', 'c'])
})
table = pa.Table.from_pandas(df)
return GeneratorStream(schema=table.schema, generator=table.to_batches())
# return RecordBatchStream(table) -> this doesnt have issue
if __name__ == '__main__':
server = FlightServer('0.0.0.0', 7688)
print(f'Starting Flight server {server.port}')
server.serve()
```
Client.py
```
from pyarrow.flight import Location, FlightClient
location = Location.for_grpc_tcp('0.0.0.0', 7688)
client = FlightClient(location)
from pyarrow.flight import Ticket
ticket = Ticket('')
reader = client.do_get(ticket)
reader.read_all()
```
### Component(s)
FlightRPC, Python
Contributor guide
Assessment
This issue has not been assessed yet.