apache / apache/arrow-rs

Consider sink-based Arrow IPC StreamEncoder API

Open
#10,445 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

`StreamEncoder::encode` currently returns `Vec`. This keeps the API simple and sans-IO friendly, but it requires the encoder to collect all output buffers for a batch before the caller can forward them to its destination.

### Describe the solution you'd like

An alternative API could push encoded pieces directly into a sink-like abstraction, for example:

```rust
pub fn encode_to(
&mut self,
batch: &RecordBatch,
sink: &mut S,
) -> Result<(), ArrowError>
```

### Describe alternatives you've considered

or make `StreamEncoder` generic over a sink.

This could reduce intermediate buffering and copying for callers that can consume IPC pieces immediately, such as async writers. The tradeoff is that the encoder starts to look more like a writer, and exposing a sink abstraction may make the public API more complex.

Things to evaluate:

- Whether this should be additive, such as `encode_to`, instead of replacing `encode`
- Whether the sink abstraction should be public
- Whether this preserves the intended sans-IO API boundary
- How it affects copy behavior and memory usage
- Benchmark impact compared with `encode -> Vec`

### Additional context

Follow-up from #10277 and review discussion:
https://github.com/apache/arrow-rs/pull/10277#discussion_r3651419972

Contributor guide

Open the contributing guide

Research direction

Start by locating the StreamEncoder::encode entry point and reviewing the follow-up discussion in #10277. Compare the current Vec behavior with the proposed sink-based alternatives, then assess API boundary, copying, memory usage, and benchmark impact. Done means reaching a decision on the API shape and documenting or implementing the agreed direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.