[Feature Request]: Customize MAX_BATCH_SIZE in DoFn.process_batch
Open
core
new feature
P2
python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
I'm using a tool in my beam template that benefits largely from batching. In general, the more the better. I noticed that my `DoFn` subclass's `process_batch` is only ever getting 4096 items. This is apparently [hard coded](https://github.com/apache/beam/blob/eed1713b5b8dc8e83ffb1fa43a4d0564083a23ce/sdks/python/apache_beam/runners/worker/operations.py#L262) and can't be changed.
Is there something special about 4096? If not, how difficult would it be to allow users to specify ideal batch sizes?
### Issue Priority
Priority: 3
### Issue Component
Component: sdk-py-core
Contributor guide
Assessment
This issue has not been assessed yet.