[Python][C++] Slow table slice when the table has many chunks
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When the pyarrow table has many chunks, the slice/take function is slow as demonstrated with the following code:
```
import sys
import time
import numpy as np
import pyarrow as pa
batch_size = 1024
batches = []
for _ in range(8555):
batch = {}
for i in range(10):
batch[str(i)] = np.array([j for j in range(batch_size)])
batches.append(pa.Table.from_pydict(batch))
block = pa.concat_tables(batches, promote=True)
# Without the below line, the time is 345s and with it, the time is 0.07s.
# block = block.combine_chunks()
start = time.perf_counter()
while block.num_rows > batch_size:
block.slice(0, batch_size)
block = block.slice(batch_size, block.num_rows - batch_size)
duration = time.perf_counter() - start
print(f"Duration: {duration}")
```
Several questions:
- Is this slice slowness expected when a table has many chunks?
- Is there a way to tell pyarrow.concat_tables to return a table with a single chunk so I can avoid an extra copy by calling combine_chunks()?
### Component(s)
Python
Contributor guide
Research direction
Start by running the Python benchmark in the issue and comparing slice/take timings with and without combine_chunks(). Then trace the table slicing path across the Python and C++ implementations, using the chunk count and reported timings to identify the relevant code path. Done means an agreed, tested resolution to the many-chunk performance problem and an answer to whether concat_tables can avoid the extra copy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100