Saving/loading dictionary type using pyarrow is broken
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When defining a column as dictionary from int32 → binary16, saving that as parquet, and reading it back – the schema is not the same as the one written, see example:
```
from tempfile import NamedTemporaryFile
import pyarrow as pa
import pyarrow.parquet as pq
if __name__ == "__main__":
z = b"\0" * 16
B16 = pa.binary(16)
D32dict = pa.dictionary(pa.int32(), B16)
tbl = pa.Table.from_arrays([[z, z, z, z]], names=["clmn"])
tbl = tbl.set_column(0, "clmn", tbl["clmn"].cast(D32dict))
assert (
tbl.schema.field("clmn").type == D32dict
), f"{tbl.schema.field('clmn').type} ≠ {D32dict}"
with NamedTemporaryFile() as fn:
pq.write_table(tbl, fn)
t = pq.read_table(fn)
assert (
t.schema.field("clmn").type == D32dict
), f"schema of table read is broken: {t.schema.field('clmn').type} ≠ {D32dict}"
```
### Component(s)
Parquet, Python
Contributor guide
Assessment
This issue has not been assessed yet.