apache / apache/beam

Inconsistent behavior on the functioning of the dataflow templates?

Open
#18,446 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

0
down vote
favorite
When i create a dataflow template, the characteristics of Runtime parameters are not persisted in the template file. At runtime, if i try to pass a value for this parameter, i take a 400 error

I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).

My parameters are the following :

```

trait MyParameters extends PipelineOptions {

def getInput: ValueProvider[String]
def setInput(value:
ValueProvider[String]): Unit
}

```

They are registred with this code

```

val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
implicit
val (sc, args) = ContextAndArgs(cmdlineArgs)

```

To create the template i call sbt with this parameters :

```

run-main jobs.XmlImportJob --runner=DataflowRunner --project=MyProject --templateLocation=gs://myBucket/XmlImportTemplate
--tempLocation=gs://myBucket/staging --instance=myInstance

```

If i pass explicitly \--input, it becomes a StaticValue instead of RuntimeValue, and this time, i can see it in the template file.

The template is called from a google function watching a bucket storage (inspired from https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/) :

```

...
dataflow.projects.templates.create({
projectId: projectId,
resource:
{
parameters: {
input: `gs://${file.bucket}/${file.name}`

},
jobName: jobs[job].name,
gcsPath: 'gs://MyBucket/MyTemplate'

}
}
...

```

The 400 error :

```

problem running dataflow template, error was: { Error: (109c1c52dc52fec7): The workflow could not be
created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: ['input' (perhaps you meant 'runner')]
at Request._callback (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
at Request.self.callback (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22)
at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at Request. (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code: 400, errors: [ { message:
'(109c1c52dc52fec7): The workflow could not be created. Causes: (109c1c52dc52fb8e): Found unexpected
parameters: [\'input\' (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ]
}

```

Imported from Jira [BEAM-2470](https://issues.apache.org/jira/browse/BEAM-2470). Original Jira may contain additional context.
Reported by: dgouyette.

Contributor guide

Open the contributing guide

Research direction

Start with the shown PipelineOptionsFactory registration, ContextAndArgs setup, and DataflowRunner template invocation. Reproduce the template with the provided sbt command, then invoke it through the templates.create request with input. Done means the runtime input parameter is persisted and accepted rather than rejected as unexpected.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, scala
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.