apache / apache/arrow-rs

Avoid repeated heap allocations and buffer copies in IPC writer

Open
#9,835 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

## Description

When writing IPC data using `StreamWriter` or `FileWriter`, the current implementation performs repeated heap allocations and full buffer copies for every record batch, even when writing batches with identical schema and structure.

This leads to unnecessary latency overhead, especially in high-frequency batch writes and streaming pipelines.

---

## Root Cause

Currently in `arrow-ipc/src/writer.rs`, the writer path is structured as:

```text
RecordBatch
→ encode() → EncodedData
→ write_message()
```

The key issue is that `EncodedData` owns its buffers:

```rust
pub struct EncodedData {
pub ipc_message: Vec,
pub arrow_data: Vec,
}
```

This forces:

* allocation of new buffers per batch
* copying of flatbuffer data into `Vec`
* destruction of all intermediate buffers after each write

---

## Current Behavior

For every batch, the following occurs:

```text
1. Build FlatBuffer (fbb)
2. Copy it → ipc_message.to_vec() (Full Copy)
3. Allocate arrow_data Vec
4. Allocate metadata vectors
5. Return EncodedData (owned)
6. write_message() writes data
7. All buffers dropped
```

### Implications

* repeated heap allocations
* repeated memory growth/reallocation
* full flatbuffer copy per batch
* memory churn (alloc → free → alloc)

---

## Proposed Solution

For repeated batch writes, the writer should ideally, without any nightly APIs or unsafe code:

```text
1. Reuse FlatBufferBuilder
2. Reuse arrow_data buffer
3. Reuse metadata vectors
4. Avoid copying flatbuffer data
5. Write directly from existing buffers
```
cc @alamb and @etseidl

Contributor guide

Open the contributing guide

Research direction

Start in arrow-ipc/src/writer.rs and trace StreamWriter and FileWriter through EncodedData and write_message(). Examine how the FlatBufferBuilder, ipc_message, arrow_data, and metadata vectors are created and consumed for each batch. Done means repeated writes reuse the relevant buffers and avoid the described full copies and per-batch allocations without nightly APIs or unsafe code.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.