apache / apache/arrow

[C++][Python] pyarrow.ChunkedArray.combine_chunks is slow

Open
#37,318 5 comments 1 reaction 1 assignee Claimed by @spenczar View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Chunked arrays are common in Pyarrow, since Tables are build on chunked arrays. Working with the underlying data requires combining chunks.

Combining chunks is surprisingly slow. In my application, I need to work with actual arrays, but my code spends a significant amount of its time calling combine_chunks even when there is just one chunk.

I have some informal benchmarking I did in an IPython session, which shows an improvement from ~150 microseconds to ~300 nanoseconds for an array of 1 million floats:

```python
In [1]: import pyarrow as pa

In [2]: import numpy as np

In [3]: data = pa.array(np.random.random(1_000_000))

In [4]: chunked_array = pa.chunked_array([data])

In [5]: %timeit chunked_array.combine_chunks()
154 µs ± 1.66 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [6]: def fast_combine_chunks(chunked_array):
...: if chunked_array.num_chunks == 1:
...: return chunked_array.chunk(0)
...: return chunked_array.combine_chunks()
...:

In [7]: %timeit fast_combine_chunks(chunked_array)
290 ns ± 2.24 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.