apache / apache/arrow

[Python] RecordBatch.serialize() should support writing into a pre-allocated buffer

Open
#49,285 0 comments 0 reactions 1 assignee Claimed by @rustyconover View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

When serializing many `RecordBatch` objects in a hot loop (e.g. streaming IPC to a socket, writing to shared memory), `RecordBatch.serialize()` allocates a new buffer on every call. This creates unnecessary allocation pressure when the caller already knows the required size and could reuse a single buffer across calls.

It would be useful if `serialize()` accepted an optional `buffer` parameter so callers can provide a pre-allocated mutable buffer to serialize into directly.

## Example usage

```python
import pyarrow as pa

batches = [...] # many RecordBatches with the same schema

# Pre-allocate once
size = max(pa.ipc.get_record_batch_size(b) for b in batches)
buf = pa.allocate_buffer(size)

for batch in batches:
result = batch.serialize(buffer=buf)
send_over_network(result) # result is a zero-copy slice of buf
```

New behavior

- `batch.serialize(buffer=buf)` serializes directly into the provided buffer and returns a slice of it with the exact serialized size.
- If the buffer is too small, a `ValueError` is raised with a message indicating the required vs. available size.
- If the buffer is not mutable, a `ValueError` is raised.
- When buffer is not provided, behavior is unchanged from today.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.