apache / apache/arrow

[C++][Parquet] Add WriteRecordBatchAsync to parquet writer

Open
#33,710 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

This [PR](https://github.com/apache/arrow/pull/33656) enhances the `parquet::arrow::FileWriter` to write columns in parallel by setting `ArrowWriterProperties::use_threads` to true and optionally providing an executor. As the [review comment](https://github.com/apache/arrow/pull/33656#pullrequestreview-1250353638) explains, a nested parallelism deadlock may occur if the file writer itself is running in the same executor.

To provide better usability, we can adopt the suggestion from @westonpace to add a `WriteRecordBatchAsync` method that calls `ParallelForAsync` and returns the future. This can then be safely called in parallel, even by thread pool threads (assuming they don't block on that future but wrap it up into a higher level `AllComplete` call later). The `WriteRecordBatch` method could then just return `WriteRecordBatchAsync(...).status()`.

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Research direction

Start by reading PR 33656 and its linked review comment, then inspect parquet::arrow::FileWriter's WriteRecordBatch path and the ParallelForAsync API. Done means adding the proposed WriteRecordBatchAsync future-returning method and making WriteRecordBatch delegate to it while preserving the described executor and nested-parallelism behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.