Avoid repeated heap allocations and buffer copies in IPC writer
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
## Description
When writing IPC data using `StreamWriter` or `FileWriter`, the current implementation performs repeated heap allocations and full buffer copies for every record batch, even when writing batches with identical schema and structure.
This leads to unnecessary latency overhead, especially in high-frequency batch writes and streaming pipelines.
---
## Root Cause
Currently in `arrow-ipc/src/writer.rs`, the writer path is structured as:
```text
RecordBatch
→ encode() → EncodedData
→ write_message()
```
The key issue is that `EncodedData` owns its buffers:
```rust
pub struct EncodedData {
pub ipc_message: Vec,
pub arrow_data: Vec,
}
```
This forces:
* allocation of new buffers per batch
* copying of flatbuffer data into `Vec`
* destruction of all intermediate buffers after each write
---
## Current Behavior
For every batch, the following occurs:
```text
1. Build FlatBuffer (fbb)
2. Copy it → ipc_message.to_vec() (Full Copy)
3. Allocate arrow_data Vec
4. Allocate metadata vectors
5. Return EncodedData (owned)
6. write_message() writes data
7. All buffers dropped
```
### Implications
* repeated heap allocations
* repeated memory growth/reallocation
* full flatbuffer copy per batch
* memory churn (alloc → free → alloc)
---
## Proposed Solution
For repeated batch writes, the writer should ideally, without any nightly APIs or unsafe code:
```text
1. Reuse FlatBufferBuilder
2. Reuse arrow_data buffer
3. Reuse metadata vectors
4. Avoid copying flatbuffer data
5. Write directly from existing buffers
```
cc @alamb and @etseidl
Contributor guide
Research direction
Start in arrow-ipc/src/writer.rs and trace StreamWriter and FileWriter through EncodedData and write_message(). Examine how the FlatBufferBuilder, ipc_message, arrow_data, and metadata vectors are created and consumed for each batch. Done means repeated writes reuse the relevant buffers and avoid the described full copies and per-batch allocations without nightly APIs or unsafe code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100