[Python][Parquet] ParquetWriter cannot use the C++ writer's parallel column encoding
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
`parquet::arrow::FileWriter` can encode a row group's columns in parallel
(`ArrowWriterProperties::set_use_threads`, GH-33655), but only via
`NewBufferedRowGroup` + `WriteRecordBatch`. pyarrow only binds `WriteTable`,
which encodes columns serially, and `ParquetWriter` has no `use_threads`
argument, so every Parquet write from Python is single-threaded
(`write_table`, `ParquetWriter`, pandas `to_parquet`).
Measured against 23.0.1 on a 37-column flat table (18 string, 8 double, 6 bool,
5 int64), zstd, 1M rows: 2.5 s with `write_table`; 1.1 s with `use_threads` at
65,536-row groups; 0.4 s at 262,144-row groups. Rows read back identical, same
row groups and statistics.
Proposed: `ParquetWriter(..., use_threads=False)` sets
`ArrowWriterProperties::set_use_threads`, and `write_table` then writes one
buffered row group per `row_group_size` rows through `WriteRecordBatch`, so the
file layout is unchanged and nothing changes unless the flag is passed.
Python-side only, ~80 lines plus tests; patch ready. The alternative would be a
`use_threads`-style option on C++ `WriteTable` so the dataset writer gets it
too. Happy to send either; which shape do you prefer?
### Component(s)
Parquet
Contributor guide
Assessment
This issue has not been assessed yet.