apache / apache/arrow

write_table does not respect indices dtype

Open
#36,589 5 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When I am working with a table that has a field of dictionary type, I try to store it as a parquet using [write_table](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html) and then load it using [read_table](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html) I noticed that write_table does not respect dictionary indices type. Please take a look at the reproducer below.

PyArrow: 12.0.1
Python: 3.11.1
Platform: MacOS, 13.4.1 (22F82)

Reproducer:
```python
import pyarrow as pa
import pyarrow.parquet as pq

table = pa.Table.from_arrays([["a", "b", None, "d"]], schema=pa.schema([pa.field("random",pa.dictionary(pa.int8(), pa.string()))]))

"""
pyarrow.Table
random: dictionary
----
random: [ -- dictionary:
["a","b","d"] -- indices:
[0,1,null,2]]
"""

pq.write_table(table, "example.parquet")
table_cpy = pq.read_table("example.parquet")

"""
pyarrow.Table
random: dictionary
----
random: [ -- dictionary:
["a","b","d"] -- indices:
[0,1,null,2]]
"""
```

It seems that the root cause is the wrong schema for the table in parquet.

```python
>>> pq.read_schema("example.parquet")
random: dictionary
```

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.