apache / apache/arrow

[Python] Get size of IPC File ahead of time

Open
#42,033 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Status: needs champion Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

TLDR: Is there any easy way of predicting the size of an IPC file ahead-of-time, before serialization?

We are trying to use `pa.Table` objects in some PyTorch data loading workflows and would like to share the underlying memory across the various data loading workers. Right now, this is possible by memory-mapping a file like:

```python
# 1 process runs this:
buffer = pa.memory_map(fname, "wb")
with pa.ipc.new_file(buffer, table.schema) as writer:
writer.write_table(table)

# All the other processes do this:
buffer = pa.memory_map(fname, "rb")
with pa.ipc.open_file(buffer) as reader:
table = reader.read_all()
```

which lets us share the same mmap physical memory across all the different processes. The problem here is that we need to coordinate around the mmap file.

Instead, I was hoping to use either `multiprocessing.shared_memory` or just a `torch.Tensor` with `uint8` dtype as the underlying buffer here to do this "automatically" (since those can be pickled by the `ForkingPickler`) -- unfortunately, to *create* a new shared memory block, I need to request a certain memory size ahead of time, and I'm not quite sure what to pass in. I was using `table.nbytes`, but this does not always align with the size of the resulting IPC file.

(Of course, maybe the right solution is for me to just write a thin `multiprocessing.reduction.ForkingPickler.reduction` wrapper around `pa.MemoryMappedFile` to just serialize the filename instead).

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.