apache / apache/arrow-rs

AsyncArrowWriter Copies Data When Flushing Row Groups

Open
#6,309 0 comments 2 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The ArrowWriter buffers encoded data pages as `RecordBatch` are submitted, so as to keep a lid on memory usage - https://github.com/apache/arrow-rs/pull/4280.

It then calls [SerializedRowGroupWriter::append_column](https://docs.rs/parquet/latest/parquet/file/writer/struct.SerializedRowGroupWriter.html#method.append_column) to write these buffered pages. This reads the data as `Bytes` using `ChunkReader` and then writes them out to the underlying `Write`.

`AsyncArrowWriter` also uses this, however, despite `AsyncFileWriter` supporting `Bytes`, which is what the data has been buffered as, the use of the slice-based `Write` forces an unnecessary copy.

**Describe the solution you'd like**

Some mechanism to avoid needing to perform this copy. Likely this would entail replacing `W: Write` with some sort of `ChunkWriter` trait to parallel `ChunkReader`.

```
// A `std::io::Write` that supports zero-copy writing of `Bytes`
trait ChunkWriter: std::io::Write {
fn put(&mut self, buf: Bytes) -> std::io::Result<()> {
self.write_all(&buf)
}
}
```

**Describe alternatives you've considered**

We could not do this, in practice the overheads of any network IO are likely to massively dominate that of a single mempcy, but creating this to document it

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing AsyncArrowWriter through SerializedRowGroupWriter::append_column and the AsyncFileWriter path. Compare the existing ChunkReader and slice-based Write interfaces with the Bytes buffering described in the issue. Done means buffered Bytes can reach the async writer without the unnecessary copy while preserving the existing writing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.