apache / apache/beam

Apply basic fusion to Java DirectRunner to avoid keeping all intermittent results in memory

Open
#20,889 0 comments 0 reactions 0 assignees View on GitHub
direct improvement P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Current java direct runner doesn't fuse transforms into steps. Instead, it almost executes each transform one by one. It results in memory pressure when any transform is high-fanout.

We already have a simple fusion logic in Java SDK(https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java). Work remaining here might be:
* Apply such fusion into DirectRunner
* Change the DirectRunner to be able run the fused steps.

I understand that DirectRunner doesn't expect processing large volume data and changing DirectRunner execution might be a fair amount of work.

Imported from Jira [BEAM-12335](https://issues.apache.org/jira/browse/BEAM-12335). Original Jira may contain additional context.
Reported by: boyuanz.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.