apache / apache/beam

[Feature Request]: Customize MAX_BATCH_SIZE in DoFn.process_batch

Open
#24,360 4 comments 0 reactions 0 assignees View on GitHub
core new feature P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

I'm using a tool in my beam template that benefits largely from batching. In general, the more the better. I noticed that my `DoFn` subclass's `process_batch` is only ever getting 4096 items. This is apparently [hard coded](https://github.com/apache/beam/blob/eed1713b5b8dc8e83ffb1fa43a4d0564083a23ce/sdks/python/apache_beam/runners/worker/operations.py#L262) and can't be changed.

Is there something special about 4096? If not, how difficult would it be to allow users to specify ideal batch sizes?

### Issue Priority

Priority: 3

### Issue Component

Component: sdk-py-core

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.