[Python] [FlightRPC] Index with value of 0 is out-of-bounds for array of length 0
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
## Context
### Description
- When a table built by concat-ing dictionary arrays of mismatched "schema"/dictionary, the transmitted table appears to be malformed
### Reproducible code
```python
import pyarrow.flight as fl
import pyarrow as pa
import enum
class MyEnum(enum.Enum):
Foo = 0
Bar = 1
Baz = 2
schema = pa.schema({
'col': pa.dictionary(pa.int8(), pa.string())
})
def build_data() -> pa.Table:
non_empty = pa.table({
'col': pa.DictionaryArray.from_arrays(pa.array([0, 2], pa.int8()), [x.name for x in MyEnum])
}, schema=schema)
empty = pa.table({
'col': pa.DictionaryArray.from_arrays(pa.array([], pa.int8()), [])
}, schema=schema)
# If unify_dictionaries get called here, it works
return pa.concat_tables([empty, non_empty]) # .unify_dictionaries()
class Server(fl.FlightServerBase):
def do_get(self, context, ticket):
table = build_data()
_ = table['col'].to_pylist()
print('build table ', table)
# This doesn't work
return fl.RecordBatchStream(table, options=pa.ipc.IpcWriteOptions(unify_dictionaries=True))
if __name__ == '__main__':
server = Server()
client = fl.FlightClient(f'grpc://localhost:{server.port}')
client.wait_for_available()
table = client.do_get(fl.Ticket(bytes())).read_all()
try:
_ = table['col'].to_pylist()
print('got table ', table)
except Exception as e:
print(e)
server.shutdown()
```
### Expectation
- `to_pylist` succeeds
### Actual
- `to_pylist` fails with `index with value of 0 is out-of-bounds for array of length 0`
### Observation
Table before IPC
```
----
col: [ -- dictionary:
[] -- indices:
[], -- dictionary:
["Foo","Bar","Baz"] -- indices:
[0,2]]
```
Table after IPC
```
col: [ -- dictionary:
[] -- indices:
[], -- dictionary:
[] -- indices:
[0,2]]
```
I'm happy to open a PR if someone can point me to the relevant code. Thanks!
### Component(s)
FlightRPC, Python
Contributor guide
Assessment
This issue has not been assessed yet.