Support for limiting parallelism of a step
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Users may want to limit the parallelism of a step. Two classic uses cases are:
- User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
- External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on the ParDo that makes the API call.
Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey with exactly k keys will guarantee that only k elements are produced, but runners are free to break fusion in ways that each element may be processed in parallel later.
To implement this functionaltiy, I believe we need to add this support to the Beam Model.
Imported from Jira [BEAM-68](https://issues.apache.org/jira/browse/BEAM-68). Original Jira may contain additional context.
Reported by: dhalperi.
Contributor guide
Research direction
Start by reviewing the Beam Model and the original Jira issue BEAM-68, which may contain additional context. Determine how a step-level parallelism limit should be represented and enforced across runners, then verify that the model can support the stated shard-count and external-QPS use cases.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100