common-workflow-language / common-workflow-language/cwltool
A Workflow with two steps of the same ExpressionTool fails if the tool inputs an array of files with loadContents
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
Both `wf1.cwl` and `wf2.cwl` pass validation.
## Actual Behavior
`wf1.cwl` validates, while `wf2.cwl` fails validation.
`wf1.cwl` contains a single instance of an `ExpressionTool`, while `wf2.cwl` contains two instances of that same tool.
The `ExpressionTool` fails as it loads the contents of an array of Files.
If the input to the `ExpressionTool` is a File, instead of an array of Files, I get successful validation in a `wf2` condition (code not included for this non-array case).
## Workflow Code
`wf1.cwl`:
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
- id: json_files
type:
type: array
items: File
outputs:
[]
steps:
- id: run1
run: expr_tool.cwl
in:
- id: json_files
source: json_files
out:
- id: output
```
`wf2.cwl`:
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
- id: json_files
type:
type: array
items: File
outputs:
[]
steps:
- id: run1
run: expr_tool.cwl
in:
- id: json_files
source: json_files
out:
- id: output
- id: run2
run: expr_tool.cwl
in:
- id: json_files
source: json_files
out:
- id: output
```
`expr_tool.cwl`:
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
requirements:
- class: InlineJavascriptRequirement
class: ExpressionTool
inputs:
- id: json_files
type:
type: array
items: File
inputBinding:
loadContents: true
outputs:
- id: output
type:
type: array
items: string
expression: |
${
var output_array = [];
for (var i = 0; i < inputs.json_files.length; i++) {
var data = JSON.parse(inputs.json_files[i].contents);
output_array.push(data['ID']);
}
return {'output': output_array};
}
```
## Successful validation of wf1
```
(p3) [jeremiah@localhost ~]$ cwltool --validate --debug wf1.cwl
/home/jeremiah/.virtualenvs/p3/bin/cwltool 1.0.20190228155703
Resolved 'wf1.cwl' to 'file:///home/jeremiah/wf1.cwl'
wf1.cwl is valid CWL.
```
## Failed validation of wf2
```pytb
(p3) [jeremiah@localhost ~]$ cwltool --validate --debug wf2.cwl
/home/jeremiah/.virtualenvs/p3/bin/cwltool 1.0.20190228155703
Resolved 'wf2.cwl' to 'file:///home/jeremiah/wf2.cwl'
Validation exception
Traceback (most recent call last):
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 630, in __init__
toolpath_object["run"], loadingContext)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 370, in load_tool
loadingContext, workflowobj, uri)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 304, in resolve_and_validate_document
schema.validate_doc(avsc_names, processobj, document_loader, loadingContext.strict)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/schema_salad/schema.py", line 361, in validate_doc
strip_dup_lineno(bullets(anyerrors, "* ")))
schema_salad.validate.ValidationException: expr_tool.cwl:2:1: Object `expr_tool.cwl` is not valid because
tried `ExpressionTool` but
expr_tool.cwl:9:1: the `inputs` field is not valid because
expr_tool.cwl:10:5: item is invalid because
expr_tool.cwl:11:5: the `type` field is not valid because
- tried InputRecordSchema but
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7: * invalid field `items`, expected one of: 'fields',
'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputEnumSchema but
* missing required field `symbols`
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7: * invalid field `items`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputArraySchema but
expr_tool.cwl:14:7: invalid field `inputBinding`, expected one of:
'items', 'type', 'label', 'doc', 'name'
Tool definition failed initialization:
Tool definition file:///home/jeremiah/expr_tool.cwl failed validation:
expr_tool.cwl:2:1: Object `expr_tool.cwl` is not valid because
tried `ExpressionTool` but
expr_tool.cwl:9:1: the `inputs` field is not valid because
expr_tool.cwl:10:5: item is invalid because
expr_tool.cwl:11:5: the `type` field is not valid because
- tried InputRecordSchema but
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7: * invalid field `items`, expected one of: 'fields',
'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputEnumSchema but
* missing required field `symbols`
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7: * invalid field `items`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputArraySchema but
expr_tool.cwl:14:7: invalid field `inputBinding`, expected one of:
'items', 'type', 'label', 'doc', 'name'
Traceback (most recent call last):
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 630, in __init__
toolpath_object["run"], loadingContext)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 370, in load_tool
loadingContext, workflowobj, uri)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 304, in resolve_and_validate_document
schema.validate_doc(avsc_names, processobj, document_loader, loadingContext.strict)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/schema_salad/schema.py", line 361, in validate_doc
strip_dup_lineno(bullets(anyerrors, "* ")))
schema_salad.validate.ValidationException: expr_tool.cwl:2:1: Object `expr_tool.cwl` is not valid because
tried `ExpressionTool` but
expr_tool.cwl:9:1: the `inputs` field is not valid because
expr_tool.cwl:10:5: item is invalid because
expr_tool.cwl:11:5: the `type` field is not valid because
- tried InputRecordSchema but
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7: * invalid field `items`, expected one of: 'fields',
'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputEnumSchema but
* missing required field `symbols`
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7: * invalid field `items`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputArraySchema but
expr_tool.cwl:14:7: invalid field `inputBinding`, expected one of:
'items', 'type', 'label', 'doc', 'name'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/main.py", line 675, in main
tool = make_tool(uri, loadingContext)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 351, in make_tool
tool = loadingContext.construct_tool_object(processobj, loadingContext)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 55, in default_make_tool
return Workflow(toolpath_object, loadingContext)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 535, in __init__
loadingContext.prov_obj))
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 569, in make_workflow_step
return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 636, in __init__
(toolpath_object["run"], validate.indent(str(vexc))))
cwltool.errors.WorkflowException: Tool definition file:///home/jeremiah/expr_tool.cwl failed validation:
expr_tool.cwl:2:1: Object `expr_tool.cwl` is not valid because
tried `ExpressionTool` but
expr_tool.cwl:9:1: the `inputs` field is not valid because
expr_tool.cwl:10:5: item is invalid because
expr_tool.cwl:11:5: the `type` field is not valid because
- tried InputRecordSchema but
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7: * invalid field `items`, expected one of: 'fields',
'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputEnumSchema but
* missing required field `symbols`
expr_tool.cwl:12:7: * the `type` field is not valid because
the value 'array' is not a valid
enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7: * invalid field `items`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7: * invalid field `inputBinding`, expected one of:
'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5: - tried InputArraySchema but
expr_tool.cwl:14:7: invalid field `inputBinding`, expected one of:
'items', 'type', 'label', 'doc', 'name'
(p3) [jeremiah@localhost ~]$
```
## Your Environment
```
(p3) [jeremiah@localhost ~]$ cwltool --version
/home/jeremiah/.virtualenvs/p3/bin/cwltool 1.0.20190228155703
```
Contributor guide
Assessment
This issue has not been assessed yet.