DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc formats.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Cloud Dataflow has a minor history of small bugs related to various code paths expecting there to be or not be a trailing forward-slash in these location fields. The way that Beam's integration tests are set up, we are likely to only have one of these two cases tested (there is a single set of integration test pipeline options).
We should add a dedicated DataflowRunner integration test to handle this case.
Actually, we should probably canonicalize the URLs so that we only ever produce one version.
Imported from Jira [BEAM-1194](https://issues.apache.org/jira/browse/BEAM-1194). Original Jira may contain additional context.
Reported by: dhalperi.
Contributor guide
Research direction
Start at the DataflowRunner entry point and trace how gcpTempLocation, stagingLocation, and the other location fields are handled. Add coverage for both trailing-slash formats, then canonicalize the URLs so only one format is produced. Done means the dedicated DataflowRunner integration test covers both inputs and passes with consistent output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100