Parquet write freezes after writing 3900th of over 11,000 partition with pyarrow 16.1
Open
Component: Python
Type: bug
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I am using pyarrow 16.1.0 and pandas 1.5.3.
I have a data sample that has over 11,000 stock names. I am trying to write hive partition on ["First letter of ticker", "ticker", "year"].
Writing from pandas (using `to_parquet` pyarrow engine) or writing directly from pyarrow (`write_to_dataset`) both stop and freeze after having written the 3897th file (out of what should be over 11,000 parquet files). The code just hangs after creating the directory to which it should write the 3898th file.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.