apache / apache/arrow

[Python][Parquet] ParquetWriter cannot use the C++ writer's parallel column encoding

Open
#50,866 0 comments 0 reactions 1 assignee Claimed by @anniegracehu View on GitHub
Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

`parquet::arrow::FileWriter` can encode a row group's columns in parallel
(`ArrowWriterProperties::set_use_threads`, GH-33655), but only via
`NewBufferedRowGroup` + `WriteRecordBatch`. pyarrow only binds `WriteTable`,
which encodes columns serially, and `ParquetWriter` has no `use_threads`
argument, so every Parquet write from Python is single-threaded
(`write_table`, `ParquetWriter`, pandas `to_parquet`).

Measured against 23.0.1 on a 37-column flat table (18 string, 8 double, 6 bool,
5 int64), zstd, 1M rows: 2.5 s with `write_table`; 1.1 s with `use_threads` at
65,536-row groups; 0.4 s at 262,144-row groups. Rows read back identical, same
row groups and statistics.

Proposed: `ParquetWriter(..., use_threads=False)` sets
`ArrowWriterProperties::set_use_threads`, and `write_table` then writes one
buffered row group per `row_group_size` rows through `WriteRecordBatch`, so the
file layout is unchanged and nothing changes unless the flag is passed.
Python-side only, ~80 lines plus tests; patch ready. The alternative would be a
`use_threads`-style option on C++ `WriteTable` so the dataset writer gets it
too. Happy to send either; which shape do you prefer?

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.