apache / apache/arrow

Partition column type is modified after write/read

Open
#32,191 0 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Example:

 
```java

s = 100000
f = 10
data = pd.DataFrame(
np.random.rand(s * f).reshape(s, f),
columns=[f"f_{i}" for i in range(f)]
)
data['partition_col'] = np.random.randint(0, f, s)
data = pyarrow.Table.from_pandas(data)
pq.write_to_dataset(data_arrow, root_path='test_pyarrow', partition_cols=['partition_col'])
data.schema
```
outputs:
```java

data.schema
f_0: double
f_1: double
f_2: double
f_3: double
f_4: double
f_5: double
f_6: double
f_7: double
f_8: double
f_9: double
partition_col: int64
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 1456
```
After writing and reading the partition col dtype turns into:
```java

pq.ParquetDataset('test_pyarrow').read().schema
f_0: double
f_1: double
f_2: double
f_3: double
f_4: double
f_5: double
f_6: double
f_7: double
f_8: double
f_9: double
partition_col: dictionary
```
 

**Environment**: Linux, Python 3.8
**Reporter**: [Daniel Gafni](https://issues.apache.org/jira/browse/ARROW-16866)

**Note**: *This issue was originally created as [ARROW-16866](https://issues.apache.org/jira/browse/ARROW-16866). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the provided Python reproduction using pq.write_to_dataset and pq.ParquetDataset('test_pyarrow').read(), then trace how partition columns are handled during dataset writing and reading. Done means the partition_col schema remains int64 after the round trip, with a regression test covering the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.