apache / apache/beam

Native support for conditional iteration

Open
#17,979 0 comments 0 reactions 0 assignees View on GitHub
new feature P3 sdk-ideas
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Ported from: https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/50

There are a variety of use cases which would benefit from native support for conditional iteration.

For instance, http://stackoverflow.com/questions/31654421/conditional-iterations-in-google-cloud-dataflow/31659923?noredirect=1#comment51264604_31659923 asks about being able to write a loop like the following:

```

PCollection data = ...
while(needsMoreWork(data)) {
data = doAStep(data)
}

```

If there are specific use cases please let us know the details. In the future we will use this issue to post progress updates.

Imported from Jira [BEAM-106](https://issues.apache.org/jira/browse/BEAM-106). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reviewing the imported Jira issue and linked Stack Overflow discussion, then identify concrete conditional-iteration use cases and the required Beam semantics. The issue is done only when a defined native iteration design is implemented with coverage for its agreed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.