apache / apache/beam

[Task]: Improve documentation of PortableRunner for usage with Spark

Open
#23,438 0 comments 0 reactions 0 assignees View on GitHub
cross-language documentation portability runners spark task website
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What needs to happen?

Users are struggling to use the portable runner with Spark. Related Spark documentation should be updated and provide detailed instructions.

> While users will probably still prefer the classic Spark runner for most Java pipelines, the Spark portable runner enables cross-language transforms. We can start by providing instructions for plain Java, then add cross-language (Java -> Python).

See:
- https://stackoverflow.com/questions/66320831/its-possible-to-configure-the-beam-portable-runner-with-the-spark-configuration
- https://stackoverflow.com/questions/66498209/how-to-configure-beam-python-sdk-with-spark-in-a-kubernetes-environment

Also related: https://github.com/apache/beam/issues/21092

### Issue Priority

Priority: 2

### Issue Component

Component: runner-spark

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.