apache / apache/beam

Support for limiting parallelism of a step

Open
#17,963 0 comments 0 reactions 0 assignees View on GitHub
beam-model new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Users may want to limit the parallelism of a step. Two classic uses cases are:

- User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
- External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on the ParDo that makes the API call.

Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey with exactly k keys will guarantee that only k elements are produced, but runners are free to break fusion in ways that each element may be processed in parallel later.

To implement this functionaltiy, I believe we need to add this support to the Beam Model.

Imported from Jira [BEAM-68](https://issues.apache.org/jira/browse/BEAM-68). Original Jira may contain additional context.
Reported by: dhalperi.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Beam Model and the original Jira issue BEAM-68, which may contain additional context. Determine how a step-level parallelism limit should be represented and enforced across runners, then verify that the model can support the stated shard-count and external-QPS use cases.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.