apache / apache/beam

DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc formats.

Open
#18,199 0 comments 0 reactions 0 assignees View on GitHub
dataflow good first issue P3 runners tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Cloud Dataflow has a minor history of small bugs related to various code paths expecting there to be or not be a trailing forward-slash in these location fields. The way that Beam's integration tests are set up, we are likely to only have one of these two cases tested (there is a single set of integration test pipeline options).

We should add a dedicated DataflowRunner integration test to handle this case.

Actually, we should probably canonicalize the URLs so that we only ever produce one version.

Imported from Jira [BEAM-1194](https://issues.apache.org/jira/browse/BEAM-1194). Original Jira may contain additional context.
Reported by: dhalperi.

Contributor guide

Open the contributing guide

Research direction

Start at the DataflowRunner entry point and trace how gcpTempLocation, stagingLocation, and the other location fields are handled. Add coverage for both trailing-slash formats, then canonicalize the URLs so only one format is produced. Done means the dedicated DataflowRunner integration test covers both inputs and passes with consistent output.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.