[Python] Batch much smaller than `batch_size` parameter
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hello, is this expected behavior?
i.e. the size of each batch being much smaller than the `batch_size` parameter?
```py
dataset = ds.dataset(f'data/level-25', format='parquet', schema=SCHEMA)
it = dataset.to_batches(batch_size=131_072, columns=['path', 'id'])
lst = [batch.num_rows for batch in it]
>>> f'{dataset.count_rows():,}'
'439,156,760'
>>> len(dataset.files)
5
>>> len(lst)
37182
>>> import statistics
>>> min(lst), max(lst), statistics.mean(lst), statistics.median(lst)
(1, 32768, 11811.004249367974, 7356.5)
```
### Component(s)
Parquet, Python
Contributor guide
Research direction
Start by reproducing the Python example using dataset.to_batches(batch_size=131_072) and inspect how the Parquet/Python path determines batch sizes. Done means establishing whether the smaller batches are expected and, if not, identifying the failing behavior with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100