apache / apache/arrow

[C++] Performance of numeric casts

Open
#40,874 11 comments 0 reactions 0 assignees View on GitHub
Component: C++
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

A simple numeric cast in pyarrow / Arrow C++ seems to be quite a bit slower compared to numpy. Often Arrow does more work (safe casting, handling nulls, ..), but I also see this this for a plain unsafe int64 -> float64 cast of a 1d array without nulsl, where Arrow is almost 4-5x slower than numpy. For a simple operation like that, this seems an unexpected big difference.

The python code I used to compare both:

```python
import numpy as np
import pyarrow as pa

nrows = 10**5
arr_np = np.random.randn(nrows*20).astype("int64")
arr = pa.array(arr_np)

%timeit arr_np.astype("float64")
# 1.1 ms ± 2.69 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%timeit arr.cast(pa.float64(), safe=False)
# 4.7 ms ± 10.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

I see this both with my local development version as with released conda packages.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.