Apply basic fusion to Java DirectRunner to avoid keeping all intermittent results in memory
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Current java direct runner doesn't fuse transforms into steps. Instead, it almost executes each transform one by one. It results in memory pressure when any transform is high-fanout.
We already have a simple fusion logic in Java SDK(https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java). Work remaining here might be:
* Apply such fusion into DirectRunner
* Change the DirectRunner to be able run the fused steps.
I understand that DirectRunner doesn't expect processing large volume data and changing DirectRunner execution might be a fair amount of work.
Imported from Jira [BEAM-12335](https://issues.apache.org/jira/browse/BEAM-12335). Original Jira may contain additional context.
Reported by: boyuanz.
Contributor guide
Assessment
This issue has not been assessed yet.