apache / apache/arrow

[C++, Parquet] dictionary(..., large_string) type not preserved when writing to Parquet

Open
#37,875 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

(tested on pyarrow-13.0.0, Linux x64)

When writing a dictionary encoded column of `large_string` type to Parquet file and reading it back, it is read back as a plain `string` type.

Repro in Python but I see the same in C++:
```
>>> import pyarrow as pa
>>> import pyarrow.compute as pc
>>> import pyarrow.parquet as pq

>>> strings = pc.dictionary_encode(pa.array(["foo, bar, foo"], pa.large_string()))
>>> table = pa.table([strings], ["strings"])
>>> table.schema
strings: dictionary

>>> pq.write_table(table, "table.parquet")
>>> pq.read_table("table.parquet").schema
strings: dictionary
```

I'd expect to get it back as a `dictionary` type.

### Component(s)

C++, Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.