apache / apache/arrow

Parquet write freezes after writing 3900th of over 11,000 partition with pyarrow 16.1

Open
#45,516 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I am using pyarrow 16.1.0 and pandas 1.5.3.

I have a data sample that has over 11,000 stock names. I am trying to write hive partition on ["First letter of ticker", "ticker", "year"].

Writing from pandas (using `to_parquet` pyarrow engine) or writing directly from pyarrow (`write_to_dataset`) both stop and freeze after having written the 3897th file (out of what should be over 11,000 parquet files). The code just hangs after creating the directory to which it should write the 3898th file.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.