[Python]How to Implement Serialization from ArrowTable to Buffer in Zero Copy?
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the usage question you have. Please include as many useful details as possible.
How to Implement Serialization from ArrowTable to Buffer in Zero Copy?
I hope that after the zero-copy serialized write is completed, the zero-copy can also be restored from the buffer to the table.
A table may have multiple batches, which are not continuous in the memory. Therefore, the table is not limited to one buffer.
```python
def table_to_buffer(table):
start = time.time()
schema, batches = table.schema, table.to_batches()
sink = pyarrow.BufferOutputStream()
stream_writer = pyarrow.RecordBatchStreamWriter(sink, schema)
for batch in batches:
stream_writer.write_batch(batch)
buffer: pyarrow.Buffer = sink.getvalue()
print(f'cost {time.time() - start:.6f} s, size {table.nbytes / (1024 ** 2):.2f} MB, bandwidth: {table.nbytes / (1024 ** 2) / (time.time() - start):.2f} MB/s')
return buffer
def buffer_to_table(buffer):
start = time.time()
reader = pyarrow.RecordBatchStreamReader(buffer)
table = reader.read_all()
print(f'cost {time.time() - start:.6f} s, size {table.nbytes / (1024 ** 2):.2f} MB, bandwidth: {table.nbytes / (1024 ** 2) / (time.time() - start):.2f} MB/s')
return table
write: cost 2.516510 s, size 3814.70 MB, bandwidth: 1515.76 MB/s
read: cost 0.000139 s, size 3814.70 MB, bandwidth: 19559902.20 MB/s
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.