Impossible to submit Google Dataflow job from Windows machine
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Got exception when trying to submit Dataflow job from java application running on Windows machine.
> java.lang.IllegalArgumentException: Location must be local or on Cloud Storage, got C:\Users\myuser\AppData\Local\Temp\dataflow-job341260414234234.json.
> Here:
[https://github.com/apache/beam/blob/67f6cf2c7786e339758854efb1cf1230909e4f1d/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L746](https://github.com/apache/beam/blob/67f6cf2c7786e339758854efb1cf1230909e4f1d/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L746)
following check is performed: fileLocation.startsWith("/") || fileLocation.startsWith("gs://"),
which prohibits usage of any Windows-style file names.
Workaround is to provide explicit file location on GCS:
dataflowPipelineOptions.setDataflowJobFile("gs://mybucket/dataflowjob.pb");
Imported from Jira [BEAM-4380](https://issues.apache.org/jira/browse/BEAM-4380). Original Jira may contain additional context.
Reported by: kardashov.
Contributor guide
Research direction
Start at runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java around line 746 and inspect the file-location check for local and Cloud Storage paths. Reproduce submission from a Windows machine, then verify that the Windows temporary path is accepted while the existing gs:// workaround and local-path behavior remain valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100