apache / apache/beam

Worker endpoints are unset in StartWorkerRequest.

Open
#20,866 1 comment 1 reaction 0 assignees View on GitHub
bug fn-execution java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

User reported that worker endpoints are unset when attempting to start a worker in a worker pool using Kubernetes. https://stackoverflow.com/questions/66498209/how-to-configure-beam-python-sdk-with-spark-in-a-kubernetes-environment

Starting worker with command ['/opt/apache/beam/boot', '\--id=1-1', '\--logging_endpoint=', '\--artifact_endpoint=', '\--provision_endpoint=', '\--control_endpoint=']

These endpoints are expected to be set to use localhost and some random allocated port. The server is started here: https://github.com/apache/beam/blob/b101064f18e71adb81c8fe6ba7beae53e9b13b08/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L672-L697
And the StartWorkerRequest is constructed here: https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ExternalEnvironmentFactory.java

Imported from Jira [BEAM-11957](https://issues.apache.org/jira/browse/BEAM-11957). Original Jira may contain additional context.
Reported by: ibzib.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.