pyarrow.dataset.write_dataset do not preserve order
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
As described, when writing a file with `pyarrow.dataset.write_dataset`, the order is not preserved. I have tested this with both `parquet` and `csv` file format.
```python
import pyarrow.parquet as pq
import numpy as np
import pandas as pd
import pyarrow.dataset
from pathlib import Path
data_load_path = './data.parquet'
pyarrow_dataset_write_path = './pyarrow_saved_data.parquet'
data = pd.DataFrame({'col': np.arange(1e7)})
data.to_parquet(data_load_path)
# Check if data loaded with pandas and pyarrow are the same
pyarrow_dataset = pyarrow.dataset.dataset(data_load_path, format='parquet')
pyarrow_dataset_df = pyarrow_dataset.to_table().to_pandas()
print((pyarrow_dataset_df['col'] == data['col']).all()) # True
# Write with pyarrow.dataset.write_dataset
pyarrow.dataset.write_dataset(
pyarrow_dataset,
pyarrow_dataset_write_path,
format='parquet',
)
loaded_pyarrow_dataset = pyarrow.dataset.dataset(pyarrow_dataset_write_path, format='parquet')
loaded_pyarrow_dataset_df = loaded_pyarrow_dataset.to_table().to_pandas()
print((loaded_pyarrow_dataset_df['col'] == data['col']).all()) # False
print((loaded_pyarrow_dataset_df['col'] == data['col']).mean()) # 0.29
# Write with pq.write_to_dataset
pq.write_to_dataset(
pyarrow_dataset,
'x.parquet',
existing_data_behavior='delete_matching'
)
(pyarrow.dataset.dataset('x.parquet').to_table().to_pandas()['col'] == data['col']).all() # True
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.