apache / apache/arrow

[Python]How to Implement Serialization from ArrowTable to Buffer in Zero Copy?

Open
#36,766 0 comments 0 reactions 0 assignees View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

How to Implement Serialization from ArrowTable to Buffer in Zero Copy?

I hope that after the zero-copy serialized write is completed, the zero-copy can also be restored from the buffer to the table.

A table may have multiple batches, which are not continuous in the memory. Therefore, the table is not limited to one buffer.

```python
def table_to_buffer(table):
start = time.time()
schema, batches = table.schema, table.to_batches()
sink = pyarrow.BufferOutputStream()

stream_writer = pyarrow.RecordBatchStreamWriter(sink, schema)
for batch in batches:
stream_writer.write_batch(batch)

buffer: pyarrow.Buffer = sink.getvalue()
print(f'cost {time.time() - start:.6f} s, size {table.nbytes / (1024 ** 2):.2f} MB, bandwidth: {table.nbytes / (1024 ** 2) / (time.time() - start):.2f} MB/s')
return buffer

def buffer_to_table(buffer):
start = time.time()
reader = pyarrow.RecordBatchStreamReader(buffer)
table = reader.read_all()
print(f'cost {time.time() - start:.6f} s, size {table.nbytes / (1024 ** 2):.2f} MB, bandwidth: {table.nbytes / (1024 ** 2) / (time.time() - start):.2f} MB/s')
return table

write: cost 2.516510 s, size 3814.70 MB, bandwidth: 1515.76 MB/s
read: cost 0.000139 s, size 3814.70 MB, bandwidth: 19559902.20 MB/s
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.