Remove redundant zero-initialization in Arrow IPC reader hot paths
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
## Description
In `arrow-ipc/src/reader.rs`, buffers are currently allocated with `MutableBuffer::from_len_zeroed(len)` and then passed to `read_exact`, which fully overwrites the entire buffer contents.
This results in an unnecessary full memory pass:
* first zero-initialization
* then complete overwrite by the incoming data
Since `read_exact` guarantees that the provided slice is fully written on success, the initial zeroing step is redundant and can be safely avoided by allocating with capacity and setting the length before the read.
## Proposed Change
Replace patterns of the form:
```rust
let mut buf = MutableBuffer::from_len_zeroed(len);
reader.read_exact(&mut buf)?;
```
with:
```rust
let mut buf = MutableBuffer::with_capacity(len);
unsafe { buf.set_len(len) };
reader.read_exact(buf.as_slice_mut())?;
```
This removes one full memory write pass per read.
## Safety Considerations
The use of `unsafe set_len` is sound in this context because:
* the buffer is not accessed or observed between `set_len` and `read_exact`
* `read_exact` either fully initializes the buffer or returns an error
* on error, the function returns immediately and the buffer is not used
* there is no exposure of partially initialized data to safe Rust code
Crash scenarios (e.g., panic, early return, process termination) do not introduce unsoundness because:
* if execution stops before `read_exact` completes, the buffer is never observed
* there is no path where uninitialized memory is read after interruption
This invariant is local and can be maintained by ensuring no intermediate access is introduced between `set_len` and the successful completion of `read_exact`.
cc: @alamb
Contributor guide
Research direction
Start in arrow-ipc/src/reader.rs by locating MutableBuffer::from_len_zeroed allocations followed by read_exact. Review the surrounding reader paths and the stated safety invariant before making changes. Done means the redundant initialization patterns are removed without exposing partially initialized data, and the Arrow IPC reader tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100