Consider a hint ParDo.withHighFanout()
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I'm finding myself again and again suggesting users on StackOverflow to insert fusion breaks after high-fanout ParDo's.
I think we should just implement this as a hint on ParDo and MapElements transforms, like we have on GroupByKey.fewKeys() or Combine.withHotKeyFanout().
E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner that implements fusion could decide to insert a runner-specific fusion break. This somewhat sidesteps the issues in https://issues.apache.org/jira/browse/BEAM-730 and https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E because every runner can decide how to do the right thing, or is free to ignore the hint.
Imported from Jira [BEAM-1234](https://issues.apache.org/jira/browse/BEAM-1234). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start with the existing GroupByKey.fewKeys() and Combine.withHotKeyFanout() hints, then inspect the ParDo and MapElements APIs and runner fusion handling. Done means the high-fanout hint behavior is decided for these transforms and runners can either honor or ignore it as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100