[Python][Docs] Max batch size for Dataset
Open
Component: Documentation
Component: Python
Status: needs champion
Type: enhancement
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
I'm using `pyarrow.Dataset.to_table(batch_size)` and the maximum batch size seems to be hard-coded to 1048576. It'd be good to make it more flexible (in my case I'm interested in numbers like 100 million), however, if it's there for a reason, mentioning it in documentation would already be helpful
### Component(s)
Documentation, Python
Contributor guide
Assessment
This issue has not been assessed yet.