apache / apache/beam

Templates shouldn't have region option populated

Open
#18,541 0 comments 0 reactions 0 assignees View on GitHub
bug core gcp io java P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

When submitting a template in the Dataflow UI, it is necessary to specify a region.

If the template also has an erroneously set region in its PipelineOptions, then the workers are not able to communicate with the service.

If we're creating a template (i.e. template_location != null), then we shouldn't populate the region.

This is an issue in Python, and likely in JAva as well. This issue hit https://stackoverflow.com/questions/47093294/templated-dataflow-hangs-over-an-hour-at-loading-and-quits-with-an-error

Imported from Jira [BEAM-3148](https://issues.apache.org/jira/browse/BEAM-3148). Original Jira may contain additional context.
Reported by: pabloem.

Contributor guide

Open the contributing guide

Research direction

Start by tracing template submission in the Python SDK where template_location and PipelineOptions region are handled, then check the corresponding Java path. Reproduce the Dataflow template case and verify that a template with an existing region does not populate or retain that region, while normal submissions still require it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.