Inconsistent behavior on the functioning of the dataflow templates?
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
0
down vote
favorite
When i create a dataflow template, the characteristics of Runtime parameters are not persisted in the template file. At runtime, if i try to pass a value for this parameter, i take a 400 error
I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
My parameters are the following :
```
trait MyParameters extends PipelineOptions {
def getInput: ValueProvider[String]
def setInput(value:
ValueProvider[String]): Unit
}
```
They are registred with this code
```
val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
implicit
val (sc, args) = ContextAndArgs(cmdlineArgs)
```
To create the template i call sbt with this parameters :
```
run-main jobs.XmlImportJob --runner=DataflowRunner --project=MyProject --templateLocation=gs://myBucket/XmlImportTemplate
--tempLocation=gs://myBucket/staging --instance=myInstance
```
If i pass explicitly \--input, it becomes a StaticValue instead of RuntimeValue, and this time, i can see it in the template file.
The template is called from a google function watching a bucket storage (inspired from https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/) :
```
...
dataflow.projects.templates.create({
projectId: projectId,
resource:
{
parameters: {
input: `gs://${file.bucket}/${file.name}`
},
jobName: jobs[job].name,
gcsPath: 'gs://MyBucket/MyTemplate'
}
}
...
```
The 400 error :
```
problem running dataflow template, error was: { Error: (109c1c52dc52fec7): The workflow could not be
created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: ['input' (perhaps you meant 'runner')]
at Request._callback (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
at Request.self.callback (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22)
at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at Request. (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code: 400, errors: [ { message:
'(109c1c52dc52fec7): The workflow could not be created. Causes: (109c1c52dc52fb8e): Found unexpected
parameters: [\'input\' (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ]
}
```
Imported from Jira [BEAM-2470](https://issues.apache.org/jira/browse/BEAM-2470). Original Jira may contain additional context.
Reported by: dgouyette.
Contributor guide
Research direction
Start with the shown PipelineOptionsFactory registration, ContextAndArgs setup, and DataflowRunner template invocation. Reproduce the template with the provided sbt command, then invoke it through the templates.create request with input. Done means the runtime input parameter is persisted and accepted rather than rejected as unexpected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, scala
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100