apache / apache/arrow

[Python][FlightRPC] IPC error using Python GeneratorStream for tables containing Categorical / DictionaryArray

Open
#38,480 11 comments 0 reactions 0 assignees View on GitHub
Component: FlightRPC Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### pyarrow=13.0.0

Seeing the following error from the Flight client side when calling reader.read_all()
`
ArrowInvalid: IPC stream did not have the expected number (1) of dictionaries at the start of the stream
table.column_names
`

The issue is only observed when:
1. the server returns a "GeneratorStream" instead of "RecordBatchStream"
2. the table contains column of Categorical / DictionaryArray type

Server.py:
```
import pyarrow as pa
from pyarrow.flight import FlightServerBase, Location, ServerCallContext, Ticket, GeneratorStream
import pandas as pd

class FlightServer(FlightServerBase):
def __init__(self, host, port):
super().__init__(Location.for_grpc_tcp(host, port))

def do_get(self, context: ServerCallContext, ticket: Ticket):
df = pd.DataFrame.from_dict({
'col_1': pd.Categorical(['a', 'b', 'c', 'a', 'b', 'c'])
})
table = pa.Table.from_pandas(df)
return GeneratorStream(schema=table.schema, generator=table.to_batches())
# return RecordBatchStream(table) -> this doesnt have issue

if __name__ == '__main__':
server = FlightServer('0.0.0.0', 7688)
print(f'Starting Flight server {server.port}')
server.serve()
```

Client.py
```
from pyarrow.flight import Location, FlightClient
location = Location.for_grpc_tcp('0.0.0.0', 7688)
client = FlightClient(location)

from pyarrow.flight import Ticket
ticket = Ticket('')
reader = client.do_get(ticket)
reader.read_all()
```

### Component(s)

FlightRPC, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.