apache / apache/arrow-rs

Combine parallel column encoding (ArrowRowGroupWriterFactory) with async writes (AsyncArrowWriter)

Open
#9,499 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Which part is this question about**

Library API, specifically the interaction between ArrowRowGroupWriterFactory / ArrowColumnChunk (sync, parallel encoding) and AsyncArrowWriter (async, sequential encoding).

**Describe your question**
We're building a high-throughput streaming k-way merge for sorted Parquet files. The write pipeline looks like:
read (rayon decode + channel prefetch) → merge sort → parallel encode (rayon) → write to disk
We want both parallel column encoding and async disk writes. Currently the API only allows picking one.
****Path A:** Parallel encode, sync write**

```
let col_writers = rg_writer_factory.create_column_writers(rg_index)?;
let chunks: Vec = rayon::install(|| {
leaves_and_writers
.into_par_iter()
.map(|(leaf, mut col_writer)| {
col_writer.write(&leaf)?;
col_writer.close()
})
.collect()
})?;

// append_to_row_group requires sync SerializedFileWriter
let mut rg = writer.next_row_group()?;
for chunk in chunks {
chunk.append_to_row_group(&mut rg)?;
}
rg.close()?;
```

**Path B: Async write, sequential encode**

```
let mut writer = AsyncArrowWriter::try_new(file, schema, Some(props))?;
writer.write(&batch).await?;
writer.close().await?;
```

**The gap:** ArrowColumnChunk (the output of parallel encoding) can only be appended through sync SerializedFileWriter. There's no async equivalent.

**Question:**
Is there a way to combine parallel encoding with async writes

**Additional context**
Both read (decode) and write (encode) use a shared rayon pool for parallelism, the only sync bottleneck is the actual disk write inside append_to_row_group

Contributor guide

Open the contributing guide

Research direction

Start by reading ArrowRowGroupWriterFactory, ArrowColumnChunk, and AsyncArrowWriter, then compare ArrowColumnChunk::append_to_row_group with AsyncArrowWriter::write and close. Define what an API supporting parallel encoding with asynchronous disk writes would need to expose, and validate the design against the two pipeline examples in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.