apache / apache/beam

Simplify use of the Python Portable runner for Go SDK pipelines

Open
#20,534 0 comments 0 reactions 0 assignees View on GitHub
go improvement P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

It's possible to execute Go SDK pipelines on any portable Beam runner, using the "universal" runner and specifying the endpoint of the job server. However, this is inconvenient in some instances as it requires having a standing Job Management server for the runner in question.

This task is to simplify using the Python Portable Runner for arbitrary/novice Go SDK users. While for performance, its generally better to keep a job management server around so it can execute multiple jobs, this isn't required.

The goal would be to create a "python" runner for the Go SDK, which will start up the python portable runner job server, and submit a pipeline to it in Loopback mode for execution, using the "universal runner", and wait for the job to finish.

 This will give Go users access to a correct runner for testing, and allow them to develop their pipelines confidently before moving them to distributed runners like Flink, Spark, or Dataflow.

Ideally outside of some clearly indicated dependencies (and failures when they aren't present), a user should be able to import the package and specify \--runner=python, and have their pipeline execute.

The "long way" for using the Python Portable Runner with the Go SDK is on the [Go Tips page of the Dev wiki. ](https://cwiki.apache.org/confluence/display/BEAM/Go+Tips) 
The Go side runner code is in [https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners](https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners) 

The Python Portable runner entry point is here: [https://github.com/apache/beam/blob/3d296c42f9d9dbb7c2234dec325f6a5255b821ee/sdks/python/apache_beam/runners/portability/portable_runner.py](https://github.com/apache/beam/blob/3d296c42f9d9dbb7c2234dec325f6a5255b821ee/sdks/python/apache_beam/runners/portability/portable_runner.py) 

 

The simplest way for this would probably be to require users have Docker installed, and for the Beam project to publish a Docker Container image that can start up the Python Runner job server appropriately. This keeps the dependencies minimal, and start up consistent for users, and we likely can re-use the technique for other purposes. And using a similar technique would make developing new SDKs easier as well, as new SDKs can use the same infrastructure from the start.

Other approaches to solve the problem are of course welcome.

Imported from Jira [BEAM-11077](https://issues.apache.org/jira/browse/BEAM-11077). Original Jira may contain additional context.
Reported by: lostluck.

Contributor guide

Open the contributing guide

Research direction

Start with the Go runner code in sdks/go/pkg/beam/runners and the Python entry point in sdks/python/apache_beam/runners/portability/portable_runner.py; the Go Tips page describes the existing long-way setup. Define the dependency and startup behavior for a Python runner, then verify that a Go user can select --runner=python and execute a pipeline without a standing job server.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, python
Domain
backend, devtools, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.