Partition column type is modified after write/read
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Example:
```java
s = 100000
f = 10
data = pd.DataFrame(
np.random.rand(s * f).reshape(s, f),
columns=[f"f_{i}" for i in range(f)]
)
data['partition_col'] = np.random.randint(0, f, s)
data = pyarrow.Table.from_pandas(data)
pq.write_to_dataset(data_arrow, root_path='test_pyarrow', partition_cols=['partition_col'])
data.schema
```
outputs:
```java
data.schema
f_0: double
f_1: double
f_2: double
f_3: double
f_4: double
f_5: double
f_6: double
f_7: double
f_8: double
f_9: double
partition_col: int64
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 1456
```
After writing and reading the partition col dtype turns into:
```java
pq.ParquetDataset('test_pyarrow').read().schema
f_0: double
f_1: double
f_2: double
f_3: double
f_4: double
f_5: double
f_6: double
f_7: double
f_8: double
f_9: double
partition_col: dictionary
```
**Environment**: Linux, Python 3.8
**Reporter**: [Daniel Gafni](https://issues.apache.org/jira/browse/ARROW-16866)
**Note**: *This issue was originally created as [ARROW-16866](https://issues.apache.org/jira/browse/ARROW-16866). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the provided Python reproduction using pq.write_to_dataset and pq.ParquetDataset('test_pyarrow').read(), then trace how partition columns are handled during dataset writing and reading. Done means the partition_col schema remains int64 after the round trip, with a regression test covering the example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100