[Python] Legacy dataset can't roundtrip Int64 with nulls if partitioned
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
If there is partitioning and the column has nulls, Int64 columns may not round trip successfully using the legacy datasets implementation.
Simple reproduction:
```python
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
import tempfile
table = pa.table({
'x': pa.array([None, 7753285016841556620]),
'y': pa.array(['a', 'b'])
})
ds_dir = tempfile.mkdtemp()
pq.write_to_dataset(table, ds_dir, partition_cols=['y'])
table_after = ds.dataset(ds_dir).to_table()
print(table['x'])
print(table_after['x'])
assert table['x'] == table_after['x']
```
```Java
[
[
null,
7753285016841556620
]
]
[
[
null
],
[
7753285016841556992
]
]
```
**Reporter**: [Will Jones](https://issues.apache.org/jira/browse/ARROW-15725) / @wjones127
#### Related issues:
- [[Python] Long-term fate of pyarrow.parquet.ParquetDataset](https://github.com/apache/arrow/issues/25775) (is related to)
**Note**: *This issue was originally created as [ARROW-15725](https://issues.apache.org/jira/browse/ARROW-15725). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the Python reproduction using pyarrow.parquet.write_to_dataset and pyarrow.dataset, focusing on the legacy datasets implementation and partitioned Int64 data with nulls. Verify the fix by rerunning the example and confirming that table_after['x'] exactly matches table['x'], including the large integer value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100