apache / apache/arrow

[Python] Batch much smaller than `batch_size` parameter

Open
#34,758 4 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hello, is this expected behavior?
i.e. the size of each batch being much smaller than the `batch_size` parameter?
```py
dataset = ds.dataset(f'data/level-25', format='parquet', schema=SCHEMA)
it = dataset.to_batches(batch_size=131_072, columns=['path', 'id'])
lst = [batch.num_rows for batch in it]

>>> f'{dataset.count_rows():,}'
'439,156,760'
>>> len(dataset.files)
5
>>> len(lst)
37182
>>> import statistics
>>> min(lst), max(lst), statistics.mean(lst), statistics.median(lst)
(1, 32768, 11811.004249367974, 7356.5)
```

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Python example using dataset.to_batches(batch_size=131_072) and inspect how the Parquet/Python path determines batch sizes. Done means establishing whether the smaller batches are expected and, if not, identifying the failing behavior with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.