Merge, Split, Delay, and Reorder bundles in the DirectRunner
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
PCollections have no guarantees about the ordering of elements between steps
Randomly reordering, splitting and merging bundles will break user pipelines which assume some order will be maintained between steps and within a bundle. This assumption is not correct, so should break in the Direct Runner.
Imported from Jira [BEAM-344](https://issues.apache.org/jira/browse/BEAM-344). Original Jira may contain additional context.
Reported by: tgroh.
Contributor guide
Research direction
Start by reading the DirectRunner's bundle-processing paths and the linked BEAM-344 context; the issue does not name specific files or tests. Determine where bundles are merged, split, delayed, and ordered, then verify that pipelines relying on accidental ordering no longer pass under the DirectRunner.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100