apache / apache/arrow

[Python] Arrow IPC 30x slower than Numpy's, and MemoryMappedFile is slower than OSFile

Open
#44,121 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I was looking around the internet for why Arrow's IPC is essentially the speed of going to disk, and I came upon a [post on stackoverflow](https://stackoverflow.com/questions/73221409/why-pyarrow-ipc-so-slow) from 2022, and ran it on my machine. I was a bit amazed at the disparity -- below are the scripts extracted from the post.

The following script using Numpy below took ~0.02s on my machine.
```Python
import numpy as np
import time
import ctypes

from multiprocessing import sharedctypes

data = np.ones([1, 1, 544, 192], dtype=np.float32)

capacity = 1000 * 1 * 544 * 192 * 10

buffer = sharedctypes.RawArray(ctypes.c_uint8, capacity + 1)
ndarray = np.ndarray((capacity,), dtype=np.uint8, buffer=buffer)

cur_offset = 0

t = time.time()
for i in range(1000):
data = np.frombuffer(data, dtype=np.uint8)
data_size = data.shape[0]
ndarray[cur_offset:data_size + cur_offset] = data
cur_offset += data_size
e = time.time()

print(e - t)
```

The script below using PyArrow ran for 0.8s on my machine.
```Python
import numpy as np
import pyarrow as pa
import time
import os

data = np.ones((1, 1, 544, 992), dtype=np.float32)

tensor = pa.Tensor.from_numpy(data)

path = os.path.join(str("./"), 'pyarrow-tensor-ipc-roundtrip')
mmap = pa.create_memory_map(path, 5000000 * 1000)

s = time.time()
for i in range(1000):
result = pa.ipc.write_tensor(tensor, mmap)
e = time.time()

print(e - s)

output_stream = pa.BufferOutputStream()

s = time.time()
for i in range(1000):
result = pa.ipc.write_tensor(tensor, output_stream)
e = time.time()

print(e - s)
```
Surprisingly second one using `BufferOutputStream` is 2x slower than the first using mmap. I also tried replacing the path with `/dev/shm/`, which is actual memory map directory, which sped things up to 0.6s. It's as if `create_memory_map` isn't using memory mapping at all. In fact if you swap out
```
mmap = pa.create_memory_map(path, 5000000 * 1000)
```
with
```
mmap = pa.OSFile(path, 'wb')
```
you'll decrease the write time by half! What's causing this?

```
os Ubuntu 24
arrow 1.3.0
pyarrow 16.1.0
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.