apache / apache/arrow-rs

Remove redundant zero-initialization in Arrow IPC reader hot paths

Open
#9,777 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

## Description

In `arrow-ipc/src/reader.rs`, buffers are currently allocated with `MutableBuffer::from_len_zeroed(len)` and then passed to `read_exact`, which fully overwrites the entire buffer contents.

This results in an unnecessary full memory pass:

* first zero-initialization
* then complete overwrite by the incoming data

Since `read_exact` guarantees that the provided slice is fully written on success, the initial zeroing step is redundant and can be safely avoided by allocating with capacity and setting the length before the read.

## Proposed Change

Replace patterns of the form:

```rust
let mut buf = MutableBuffer::from_len_zeroed(len);
reader.read_exact(&mut buf)?;
```

with:

```rust
let mut buf = MutableBuffer::with_capacity(len);
unsafe { buf.set_len(len) };
reader.read_exact(buf.as_slice_mut())?;
```

This removes one full memory write pass per read.

## Safety Considerations

The use of `unsafe set_len` is sound in this context because:

* the buffer is not accessed or observed between `set_len` and `read_exact`
* `read_exact` either fully initializes the buffer or returns an error
* on error, the function returns immediately and the buffer is not used
* there is no exposure of partially initialized data to safe Rust code

Crash scenarios (e.g., panic, early return, process termination) do not introduce unsoundness because:

* if execution stops before `read_exact` completes, the buffer is never observed
* there is no path where uninitialized memory is read after interruption

This invariant is local and can be maintained by ensuring no intermediate access is introduced between `set_len` and the successful completion of `read_exact`.

cc: @alamb

Contributor guide

Open the contributing guide

Research direction

Start in arrow-ipc/src/reader.rs by locating MutableBuffer::from_len_zeroed allocations followed by read_exact. Review the surrounding reader paths and the stated safety invariant before making changes. Done means the redundant initialization patterns are removed without exposing partially initialized data, and the Arrow IPC reader tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.