[Python] Arrow IPC 30x slower than Numpy's, and MemoryMappedFile is slower than OSFile
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I was looking around the internet for why Arrow's IPC is essentially the speed of going to disk, and I came upon a [post on stackoverflow](https://stackoverflow.com/questions/73221409/why-pyarrow-ipc-so-slow) from 2022, and ran it on my machine. I was a bit amazed at the disparity -- below are the scripts extracted from the post.
The following script using Numpy below took ~0.02s on my machine.
```Python
import numpy as np
import time
import ctypes
from multiprocessing import sharedctypes
data = np.ones([1, 1, 544, 192], dtype=np.float32)
capacity = 1000 * 1 * 544 * 192 * 10
buffer = sharedctypes.RawArray(ctypes.c_uint8, capacity + 1)
ndarray = np.ndarray((capacity,), dtype=np.uint8, buffer=buffer)
cur_offset = 0
t = time.time()
for i in range(1000):
data = np.frombuffer(data, dtype=np.uint8)
data_size = data.shape[0]
ndarray[cur_offset:data_size + cur_offset] = data
cur_offset += data_size
e = time.time()
print(e - t)
```
The script below using PyArrow ran for 0.8s on my machine.
```Python
import numpy as np
import pyarrow as pa
import time
import os
data = np.ones((1, 1, 544, 992), dtype=np.float32)
tensor = pa.Tensor.from_numpy(data)
path = os.path.join(str("./"), 'pyarrow-tensor-ipc-roundtrip')
mmap = pa.create_memory_map(path, 5000000 * 1000)
s = time.time()
for i in range(1000):
result = pa.ipc.write_tensor(tensor, mmap)
e = time.time()
print(e - s)
output_stream = pa.BufferOutputStream()
s = time.time()
for i in range(1000):
result = pa.ipc.write_tensor(tensor, output_stream)
e = time.time()
print(e - s)
```
Surprisingly second one using `BufferOutputStream` is 2x slower than the first using mmap. I also tried replacing the path with `/dev/shm/`, which is actual memory map directory, which sped things up to 0.6s. It's as if `create_memory_map` isn't using memory mapping at all. In fact if you swap out
```
mmap = pa.create_memory_map(path, 5000000 * 1000)
```
with
```
mmap = pa.OSFile(path, 'wb')
```
you'll decrease the write time by half! What's causing this?
```
os Ubuntu 24
arrow 1.3.0
pyarrow 16.1.0
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.