[C++][Python] pyarrow.ChunkedArray.combine_chunks is slow
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Chunked arrays are common in Pyarrow, since Tables are build on chunked arrays. Working with the underlying data requires combining chunks.
Combining chunks is surprisingly slow. In my application, I need to work with actual arrays, but my code spends a significant amount of its time calling combine_chunks even when there is just one chunk.
I have some informal benchmarking I did in an IPython session, which shows an improvement from ~150 microseconds to ~300 nanoseconds for an array of 1 million floats:
```python
In [1]: import pyarrow as pa
In [2]: import numpy as np
In [3]: data = pa.array(np.random.random(1_000_000))
In [4]: chunked_array = pa.chunked_array([data])
In [5]: %timeit chunked_array.combine_chunks()
154 µs ± 1.66 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [6]: def fast_combine_chunks(chunked_array):
...: if chunked_array.num_chunks == 1:
...: return chunked_array.chunk(0)
...: return chunked_array.combine_chunks()
...:
In [7]: %timeit fast_combine_chunks(chunked_array)
290 ns ± 2.24 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.