AsyncArrowWriter Copies Data When Flushing Row Groups
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The ArrowWriter buffers encoded data pages as `RecordBatch` are submitted, so as to keep a lid on memory usage - https://github.com/apache/arrow-rs/pull/4280.
It then calls [SerializedRowGroupWriter::append_column](https://docs.rs/parquet/latest/parquet/file/writer/struct.SerializedRowGroupWriter.html#method.append_column) to write these buffered pages. This reads the data as `Bytes` using `ChunkReader` and then writes them out to the underlying `Write`.
`AsyncArrowWriter` also uses this, however, despite `AsyncFileWriter` supporting `Bytes`, which is what the data has been buffered as, the use of the slice-based `Write` forces an unnecessary copy.
**Describe the solution you'd like**
Some mechanism to avoid needing to perform this copy. Likely this would entail replacing `W: Write` with some sort of `ChunkWriter` trait to parallel `ChunkReader`.
```
// A `std::io::Write` that supports zero-copy writing of `Bytes`
trait ChunkWriter: std::io::Write {
fn put(&mut self, buf: Bytes) -> std::io::Result<()> {
self.write_all(&buf)
}
}
```
**Describe alternatives you've considered**
We could not do this, in practice the overheads of any network IO are likely to massively dominate that of a single mempcy, but creating this to document it
**Additional context**
Contributor guide
Research direction
Start by tracing AsyncArrowWriter through SerializedRowGroupWriter::append_column and the AsyncFileWriter path. Compare the existing ChunkReader and slice-based Write interfaces with the Bytes buffering described in the issue. Done means buffered Bytes can reach the async writer without the unnecessary copy while preserving the existing writing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100