[Python] RecordBatch.serialize() silently drops dictionary messages
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hi Arrow Friends,
`RecordBatch.serialize()` on a batch with dictionary encoded columns produces a single IPC record batch message containing only the indices. The dictionary values message is not included. Attempting to read the result back with `ipc.read_record_batch()` fails with `ArrowKeyError: Dictionary field not found.`
A full IPC stream via `ipc.new_stream` correctly writes schema, dictionary, and record batch messages and round-trips fine.
This is surprising because serialize() succeeds without error, producing bytes that cannot be deserialized.
I think we have some gaps in the PyArrow IPC API regarding `DictionaryMemo` and the ability to interpret `Dictionary` IPC messages.
To see the issue:
```python
import pyarrow as pa
from pyarrow import ipc
arr = pa.array(["apple", "banana", "apple", "cherry", "banana"]).dictionary_encode()
batch = pa.record_batch([arr], names=["fruit"])
# serialize() produces only a record batch message — no dictionary message
raw = batch.serialize().to_pybytes()
reader = pa.BufferReader(pa.py_buffer(raw))
while (msg := ipc.read_message(reader)) is not None:
print(msg.type)
# Output: record batch
# Round-trip fails
ipc.read_record_batch(pa.py_buffer(raw), batch.schema)
# ArrowKeyError: Dictionary field not found
```
For comparison, a full IPC stream includes the dictionary:
```python
sink = pa.BufferOutputStream()
writer = ipc.new_stream(sink, batch.schema)
writer.write_batch(batch)
writer.close()
stream = sink.getvalue().to_pybytes()
reader = pa.BufferReader(pa.py_buffer(stream))
while (msg := ipc.read_message(reader)) is not None:
print(msg.type)
# Output: schema, dictionary, record batch
```
This bug is kind of a motivating/organizing one. I'm going to be creating a series of PRs that increase the PyArrow API around dictionary messages so that it is easier for users to serialize IPC RecordBatches with dictionaries.
Rusty
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.