apache / apache/arrow

[Python] Legacy dataset can't roundtrip Int64 with nulls if partitioned

Open
#31,175 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

If there is partitioning and the column has nulls, Int64 columns may not round trip successfully using the legacy datasets implementation. 

Simple reproduction:

 ```python

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
import tempfile

table = pa.table({
'x': pa.array([None, 7753285016841556620]),
'y': pa.array(['a', 'b'])
})

ds_dir = tempfile.mkdtemp()
pq.write_to_dataset(table, ds_dir, partition_cols=['y'])

table_after = ds.dataset(ds_dir).to_table()
print(table['x'])
print(table_after['x'])
assert table['x'] == table_after['x']
```

```Java

[
[
null,
7753285016841556620
]
]
[
[
null
],
[
7753285016841556992
]
]
```

**Reporter**: [Will Jones](https://issues.apache.org/jira/browse/ARROW-15725) / @wjones127
#### Related issues:
- [[Python] Long-term fate of pyarrow.parquet.ParquetDataset](https://github.com/apache/arrow/issues/25775) (is related to)

**Note**: *This issue was originally created as [ARROW-15725](https://issues.apache.org/jira/browse/ARROW-15725). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python reproduction using pyarrow.parquet.write_to_dataset and pyarrow.dataset, focusing on the legacy datasets implementation and partitioned Int64 data with nulls. Verify the fix by rerunning the example and confirming that table_after['x'] exactly matches table['x'], including the large integer value.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.