apache / apache/arrow

[Python][C++] Slow table slice when the table has many chunks

Open
#35,126 4 comments 2 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When the pyarrow table has many chunks, the slice/take function is slow as demonstrated with the following code:

```
import sys
import time
import numpy as np
import pyarrow as pa

batch_size = 1024

batches = []
for _ in range(8555):
batch = {}
for i in range(10):
batch[str(i)] = np.array([j for j in range(batch_size)])
batches.append(pa.Table.from_pydict(batch))
block = pa.concat_tables(batches, promote=True)

# Without the below line, the time is 345s and with it, the time is 0.07s.
# block = block.combine_chunks()

start = time.perf_counter()
while block.num_rows > batch_size:
block.slice(0, batch_size)
block = block.slice(batch_size, block.num_rows - batch_size)

duration = time.perf_counter() - start
print(f"Duration: {duration}")
```

Several questions:
- Is this slice slowness expected when a table has many chunks?
- Is there a way to tell pyarrow.concat_tables to return a table with a single chunk so I can avoid an extra copy by calling combine_chunks()?

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by running the Python benchmark in the issue and comparing slice/take timings with and without combine_chunks(). Then trace the table slicing path across the Python and C++ implementations, using the chunk count and reported timings to identify the relevant code path. Done means an agreed, tested resolution to the many-chunk performance problem and an answer to whether concat_tables can avoid the extra copy.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.