apache / apache/beam

Impossible to submit Google Dataflow job from Windows machine

Closed
#18,734 1 comment 0 reactions 0 assignees View on GitHub
bug dataflow good first issue P4 runners windows
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Got exception when trying to submit Dataflow job from java application running on Windows machine.
> java.lang.IllegalArgumentException: Location must be local or on Cloud Storage, got C:\Users\myuser\AppData\Local\Temp\dataflow-job341260414234234.json.
>  Here:

[https://github.com/apache/beam/blob/67f6cf2c7786e339758854efb1cf1230909e4f1d/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L746](https://github.com/apache/beam/blob/67f6cf2c7786e339758854efb1cf1230909e4f1d/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L746)

following check is performed:  fileLocation.startsWith("/") || fileLocation.startsWith("gs://"),

which prohibits usage of any Windows-style file names.

Workaround is to provide explicit file location on GCS:

dataflowPipelineOptions.setDataflowJobFile("gs://mybucket/dataflowjob.pb");

 

Imported from Jira [BEAM-4380](https://issues.apache.org/jira/browse/BEAM-4380). Original Jira may contain additional context.
Reported by: kardashov.

Contributor guide

Open the contributing guide

Research direction

Start at runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java around line 746 and inspect the file-location check for local and Cloud Storage paths. Reproduce submission from a Windows machine, then verify that the Windows temporary path is accepted while the existing gs:// workaround and local-path behavior remain valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.