Consider sink-based Arrow IPC StreamEncoder API
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Is your feature request related to a problem or challenge?
`StreamEncoder::encode` currently returns `Vec`. This keeps the API simple and sans-IO friendly, but it requires the encoder to collect all output buffers for a batch before the caller can forward them to its destination.
### Describe the solution you'd like
An alternative API could push encoded pieces directly into a sink-like abstraction, for example:
```rust
pub fn encode_to(
&mut self,
batch: &RecordBatch,
sink: &mut S,
) -> Result<(), ArrowError>
```
### Describe alternatives you've considered
or make `StreamEncoder` generic over a sink.
This could reduce intermediate buffering and copying for callers that can consume IPC pieces immediately, such as async writers. The tradeoff is that the encoder starts to look more like a writer, and exposing a sink abstraction may make the public API more complex.
Things to evaluate:
- Whether this should be additive, such as `encode_to`, instead of replacing `encode`
- Whether the sink abstraction should be public
- Whether this preserves the intended sans-IO API boundary
- How it affects copy behavior and memory usage
- Benchmark impact compared with `encode -> Vec`
### Additional context
Follow-up from #10277 and review discussion:
https://github.com/apache/arrow-rs/pull/10277#discussion_r3651419972
Contributor guide
Research direction
Start by locating the StreamEncoder::encode entry point and reviewing the follow-up discussion in #10277. Compare the current Vec behavior with the proposed sink-based alternatives, then assess API boundary, copying, memory usage, and benchmark impact. Done means reaching a decision on the API shape and documenting or implementing the agreed direction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100