common-workflow-language / common-workflow-language/cwltool

Refering to inputs parameter with valueFrom within step triggers evaluation error

Open
#1,203 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
Assume that in a `WorkflowStep` you need to refer to a workflow (main) parameter named e.g. `main_input`. Then the ad-hoc idiom (as documented [here](https://www.biostars.org/p/378279/)) seems to be
```
inputs:
main_input: string
steps:
main_step:
run: ...
in:
sub_input:
source: main_input # Ref: https://www.biostars.org/p/378279/
valueFrom: "$(inputs.main_input)"
```
Since this idiom seems to be effective in some context, we expect it to handle over the value of `main_input` parameter to the specified nested Worflow.

## Actual Behavior
The above idiom triggers the following error message
```
ERROR [step main_step] Cannot make job: Expression evaluation error:
Expecting value: line 1 column 1 (char 0)
script was:
01 "use strict";
02 var inputs = {
03 "sub_input": "some-string-from-main-input"
04 };
05 var self = "some-string-from-main-input";
06 var runtime = {
07 "tmpdir": null,
08 "outdir": null
09 };
10 (function(){return ((inputs.main_input));})()
stdout was: 'undefined'
stderr was: ''
```

## Workflow Code
For convenience all the following material is provided by [this valuefrom_problem.zip archive](https://github.com/common-workflow-language/cwltool/files/3680097/valuefrom_problem.zip): refer to the included `Readme.md` for instructions.

Consider the [echo example of the user_guide](https://www.commonwl.org/user_guide/02-1st-example/index.html), that is relabeled to
_subworkflow.cwl_
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.1
class: CommandLineTool
baseCommand: echo
inputs:
sub_input:
type: string
inputBinding:
position: 1
outputs: []
```
and that, as expected, validates and runs smoothly.

Now consider the following workflow
_main.cwl_
```
#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: Workflow

requirements:
SubworkflowFeatureRequirement: {}
StepInputExpressionRequirement: {}
InlineJavascriptRequirement: {}

inputs:
main_input: string

steps:
main_step:
run: subworkflow.cwl
in:
sub_input:
source: main_input # Ref: https://www.biostars.org/p/378279/
valueFrom: "$(inputs.main_input)"
out: []

outputs: []
```
and the following input file
_main-input.yml _
```
main_input: some-string-from-main-input
```
Although `main.cwl` validates (`cwl-runner --validate main.cwl`) properly when running it one gets
```
$ cwl-runner main.cwl main-input.yml
INFO /private/tmp/valuefrom_problem/venv/bin/cwl-runner 1.0.20190915164430
INFO Resolved 'main.cwl' to 'file:///private/tmp/valuefrom_problem/main.cwl'
INFO [workflow ] start
INFO [workflow ] starting step main_step
ERROR [step main_step] Cannot make job: Expression evaluation error:
Expecting value: line 1 column 1 (char 0)
script was:
01 "use strict";
02 var inputs = {
03 "sub_input": "some-string-from-main-input"
04 };
05 var self = "some-string-from-main-input";
06 var runtime = {
07 "tmpdir": null,
08 "outdir": null
09 };
10 (function(){return ((inputs.main_input));})()
stdout was: 'undefined'
stderr was: ''

INFO [workflow ] completed permanentFail
{}
WARNING Final process status is permanentFail
(venv) EBO_PC(eboix):
```

## Full Traceback
```
$ cwl-runner --debug main.cwl main-input.yml
INFO /private/tmp/valuefrom_problem/venv/bin/cwl-runner 1.0.20190915164430
INFO Resolved 'main.cwl' to 'file:///private/tmp/valuefrom_problem/main.cwl'
DEBUG [workflow ] initialized from file:///private/tmp/valuefrom_problem/main.cwl
INFO [workflow ] start
DEBUG [workflow ] {
"main_input": "some-string-from-main-input"
}
INFO [workflow ] starting step main_step
DEBUG [job step main_step] job input {
"file:///private/tmp/valuefrom_problem/main.cwl#main_step/sub_input": "some-string-from-main-input"
}
ERROR [step main_step] Cannot make job: Expression evaluation error:
Expecting value: line 1 column 1 (char 0)
script was:
01 "use strict";
02 var inputs = {
03 "sub_input": "some-string-from-main-input"
04 };
05 var self = "some-string-from-main-input";
06 var runtime = {
07 "tmpdir": null,
08 "outdir": null
09 };
10 (function(){return ((inputs.main_input));})()
stdout was: 'undefined'
stderr was: ''

DEBUG
Traceback (most recent call last):
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/sandboxjs.py", line 378, in execjs
return cast(JSON, json.loads(stdout))
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/expression.py", line 304, in do_eval
strip_whitespace=strip_whitespace)
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/expression.py", line 238, in interpolate
debug=debug, js_console=js_console)
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/expression.py", line 203, in evaluator
debug=debug, js_console=js_console)
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/sandboxjs.py", line 382, in execjs
err, fn_linenum(), stdout, stderr)), err)
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/future/utils/__init__.py", line 400, in raise_from
exec(execstr, myglobals, mylocals)
File "", line 1, in
cwltool.sandboxjs.JavascriptException: Expecting value: line 1 column 1 (char 0)
script was:
01 "use strict";
02 var inputs = {
03 "sub_input": "some-string-from-main-input"
04 };
05 var self = "some-string-from-main-input";
06 var runtime = {
07 "tmpdir": null,
08 "outdir": null
09 };
10 (function(){return ((inputs.main_input));})()
stdout was: 'undefined'
stderr was: ''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/workflow.py", line 477, in job
for newjob in step.iterable:
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/workflow.py", line 399, in try_make_job
inputobj = postScatterEval(inputobj)
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/workflow.py", line 368, in postScatterEval
return {k: valueFromFunc(k, v) for k, v in io.items()}
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/workflow.py", line 368, in
return {k: valueFromFunc(k, v) for k, v in io.items()}
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/workflow.py", line 365, in valueFromFunc
timeout=runtimeContext.eval_timeout)
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/cwltool/expression.py", line 307, in do_eval
raise_from(WorkflowException("Expression evaluation error:\n%s" % Text(e)), e)
File "/private/tmp/valuefrom_problem/venv/lib/python3.7/site-packages/future/utils/__init__.py", line 400, in raise_from
exec(execstr, myglobals, mylocals)
File "", line 1, in
cwltool.errors.WorkflowException: Expression evaluation error:
Expecting value: line 1 column 1 (char 0)
script was:
01 "use strict";
02 var inputs = {
03 "sub_input": "some-string-from-main-input"
04 };
05 var self = "some-string-from-main-input";
06 var runtime = {
07 "tmpdir": null,
08 "outdir": null
09 };
10 (function(){return ((inputs.main_input));})()
stdout was: 'undefined'
stderr was: ''

INFO [workflow ] completed permanentFail
DEBUG [workflow ] {}
{}
WARNING Final process status is permanentFail
```

## Your Environment
* cwltool version: 1.0.20190915164430

## Notes
- Notice that the main parameter value is probably properly extracted from the main "source" as illustrated by the line
```
03 "sub_input": "some-string-from-main-input"
```
"Expression evaluation error".
- Could this issue be somehow related with issue #1147 ?
- If the above workflow code were to be correct then even a kludgy workaround (a more "complicated" valueFrom expression?) would be greatly appreciated :-)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.