Combine parallel column encoding (ArrowRowGroupWriterFactory) with async writes (AsyncArrowWriter)
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Which part is this question about**
Library API, specifically the interaction between ArrowRowGroupWriterFactory / ArrowColumnChunk (sync, parallel encoding) and AsyncArrowWriter (async, sequential encoding).
**Describe your question**
We're building a high-throughput streaming k-way merge for sorted Parquet files. The write pipeline looks like:
read (rayon decode + channel prefetch) → merge sort → parallel encode (rayon) → write to disk
We want both parallel column encoding and async disk writes. Currently the API only allows picking one.
****Path A:** Parallel encode, sync write**
```
let col_writers = rg_writer_factory.create_column_writers(rg_index)?;
let chunks: Vec = rayon::install(|| {
leaves_and_writers
.into_par_iter()
.map(|(leaf, mut col_writer)| {
col_writer.write(&leaf)?;
col_writer.close()
})
.collect()
})?;
// append_to_row_group requires sync SerializedFileWriter
let mut rg = writer.next_row_group()?;
for chunk in chunks {
chunk.append_to_row_group(&mut rg)?;
}
rg.close()?;
```
**Path B: Async write, sequential encode**
```
let mut writer = AsyncArrowWriter::try_new(file, schema, Some(props))?;
writer.write(&batch).await?;
writer.close().await?;
```
**The gap:** ArrowColumnChunk (the output of parallel encoding) can only be appended through sync SerializedFileWriter. There's no async equivalent.
**Question:**
Is there a way to combine parallel encoding with async writes
**Additional context**
Both read (decode) and write (encode) use a shared rayon pool for parallelism, the only sync bottleneck is the actual disk write inside append_to_row_group
Contributor guide
Research direction
Start by reading ArrowRowGroupWriterFactory, ArrowColumnChunk, and AsyncArrowWriter, then compare ArrowColumnChunk::append_to_row_group with AsyncArrowWriter::write and close. Define what an API supporting parallel encoding with asynchronous disk writes would need to expose, and validate the design against the two pipeline examples in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100