apache / apache/beam

Create an Executable Graph representation in the Universal Local Runner

Open
#18,671 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 portability runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

This represents the entirety of a Pipeline, divided into executable stages which are executed in either the runner or within a user container. This representation must include the ports over which the SDK harness communicates to the runner.

The construction of this graph likely includes most of the nodes present within the Runner API graph (PTransform and PCollection), but injects additional nodes to represent a remote read or write between harnesses.

Simple fusion (naive producer/consumer and sibling fusion) also should be performed here.

This will also require rewriting some boundary coders (e.g. runner -\> SDK Harness and vice versa) to be agnostic to the language of the runner harness, likely by converting into length-prefixed bytes.

Imported from Jira [BEAM-3337](https://issues.apache.org/jira/browse/BEAM-3337). Original Jira may contain additional context.
Reported by: tgroh.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Runner API graph concepts named in the issue, especially PTransform, PCollection, SDK harnesses, and boundary coders. Done means the Universal Local Runner has an executable stage graph with runner and container execution boundaries, SDK communication ports, simple fusion, and language-agnostic boundary encoding.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.