apache / apache/arrow

[Python][Parquet] Data corruption when writing partitioned parquet dataset with nested lists depending on number of rows

Open
#45,765 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

**Environment**
* Windows 11 (build 22631.4890)
* pyarrow 19.0.1
* Python 3.11.10

**Issue Description**
Data in nested list columns can be corrupted when writing partitioned parquet datasets, seemingly depending on how many rows the dataset has.

**Reproducible example**

```python
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

filename = 'repro.parquet'

n_rows = 11_000 # works with 10_000 rows
n_partitions = 7
n_nested_size = 100_000

lst = list(range(n_nested_size))
r = range(n_rows)

table = pa.table({
'id': list(r),
'partition': [i % n_partitions for i in r],
'nested': [np.random.permutation(lst) for _ in r],
})

pq.write_to_dataset(table, filename, partition_cols=['partition']) # works without partitioning

df = pq.read_pandas(filename)
for i in r:
assert len(set(df['nested'][i])) == n_nested_size # assertion error is triggered
```

**Findings**
- Issue does not appear if you only choose 10'000 rows in the example above
- Issue does not appear without partitioning
- Issue does not appear with pyarrow 10.0.1
- Issue also happens in pyarrow 11.0.0, 15.0.1, 16.1.0, 17.0.0 and 18.1.0

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.