apache / apache/arrow

Saving/loading dictionary type using pyarrow is broken

Open
#44,350 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When defining a column as dictionary from int32 → binary16, saving that as parquet, and reading it back – the schema is not the same as the one written, see example:

```
from tempfile import NamedTemporaryFile

import pyarrow as pa
import pyarrow.parquet as pq

if __name__ == "__main__":
z = b"\0" * 16
B16 = pa.binary(16)
D32dict = pa.dictionary(pa.int32(), B16)
tbl = pa.Table.from_arrays([[z, z, z, z]], names=["clmn"])
tbl = tbl.set_column(0, "clmn", tbl["clmn"].cast(D32dict))
assert (
tbl.schema.field("clmn").type == D32dict
), f"{tbl.schema.field('clmn').type} ≠ {D32dict}"
with NamedTemporaryFile() as fn:
pq.write_table(tbl, fn)
t = pq.read_table(fn)
assert (
t.schema.field("clmn").type == D32dict
), f"schema of table read is broken: {t.schema.field('clmn').type} ≠ {D32dict}"
```

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.