[Python] Get size of IPC File ahead of time
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the usage question you have. Please include as many useful details as possible.
TLDR: Is there any easy way of predicting the size of an IPC file ahead-of-time, before serialization?
We are trying to use `pa.Table` objects in some PyTorch data loading workflows and would like to share the underlying memory across the various data loading workers. Right now, this is possible by memory-mapping a file like:
```python
# 1 process runs this:
buffer = pa.memory_map(fname, "wb")
with pa.ipc.new_file(buffer, table.schema) as writer:
writer.write_table(table)
# All the other processes do this:
buffer = pa.memory_map(fname, "rb")
with pa.ipc.open_file(buffer) as reader:
table = reader.read_all()
```
which lets us share the same mmap physical memory across all the different processes. The problem here is that we need to coordinate around the mmap file.
Instead, I was hoping to use either `multiprocessing.shared_memory` or just a `torch.Tensor` with `uint8` dtype as the underlying buffer here to do this "automatically" (since those can be pickled by the `ForkingPickler`) -- unfortunately, to *create* a new shared memory block, I need to request a certain memory size ahead of time, and I'm not quite sure what to pass in. I was using `table.nbytes`, but this does not always align with the size of the resulting IPC file.
(Of course, maybe the right solution is for me to just write a thin `multiprocessing.reduction.ForkingPickler.reduction` wrapper around `pa.MemoryMappedFile` to just serialize the filename instead).
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.