common-workflow-language / common-workflow-language/cwltool

Error on nested record schema when not a type union

Open
#1,296 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
Tool should allow for nested record schemas

## Actual Behavior
If nested record schema is not listed as a type union, an exception is thrown. Validating externally to the cwltool passes validation as expected, but fails when run with cwltool.

## Workflow Code
```
class: CommandLineTool
cwlVersion: v1.1
label: Something tool
doc: Tool that does something

baseCommand: echo
arguments: [ $(inputs.a.c.d) ]

inputs:
a:
type:
label: A record schema
type: record
fields:
b:
doc: This is inputs.a.b
label: Some integer
type: int
c:
label: This is inputs.a.c
type:
# causes error
type: record
label: A nested record schema
fields:
d: float
outputs: []
```

## Job Input

```
a:
b: 1
c:
d: 1.0
```

## Full Traceback
```pytb
$  cwltool --debug nested_record.cwl nested_record_job.yml
INFO /home/cdodd/.local/bin//cwltool 3.0.20200324120055
INFO Resolved 'nested_record.cwl' to 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/nested_record.cwl'
ERROR Tool definition failed validation:
nested_record.cwl:9:1: 'name'
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/process.py", line 641, in __init__
schema.make_valid_avro(self.inputs_record_schema, {}, set()),
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/schema.py", line 533, in make_valid_avro
items[field], alltypes, found, union=True
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/schema.py", line 541, in make_valid_avro
ret.append(make_valid_avro(i, alltypes, found, union=union))
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/schema.py", line 533, in make_valid_avro
items[field], alltypes, found, union=True
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/schema.py", line 533, in make_valid_avro
items[field], alltypes, found, union=True
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/schema.py", line 541, in make_valid_avro
ret.append(make_valid_avro(i, alltypes, found, union=union))
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/schema.py", line 533, in make_valid_avro
items[field], alltypes, found, union=True
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/schema.py", line 527, in make_valid_avro
if items["name"] in found:
File "/home/cdodd/.local/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 753, in __getitem__
return ordereddict.__getitem__(self, key)
KeyError: 'name'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/main.py", line 942, in main
tool = make_tool(uri, loadingContext)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/load_tool.py", line 414, in make_tool
tool = loadingContext.construct_tool_object(processobj, loadingContext)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/workflow.py", line 57, in default_make_tool
return command_line_tool.CommandLineTool(toolpath_object, loadingContext)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 333, in __init__
super(CommandLineTool, self).__init__(toolpath_object, loadingContext)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/process.py", line 644, in __init__
schema.convert_to_dict(self.inputs_record_schema), self.names
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/sourceline.py", line 305, in __exit__
raise self.makeError(str(exc_value)) from exc_value
schema_salad.exceptions.ValidationException: nested_record.cwl:9:1: 'name'
```

## Workflow Code (no error when nested record schema specifies a type union instead)
```
class: CommandLineTool
cwlVersion: v1.1
label: Something tool
doc: Tool that does something
baseCommand: echo
arguments: [ $(inputs.a.c.d) ]

inputs:
a:
type:
label: A record schema
type: record
fields:
b:
doc: This is inputs.a.b
label: Some integer
type: int

c:
label: This is inputs.a.c
type:
# type union with only one type (ok)
- type: record
label: A nested record schema
fields:
d: float

outputs: []
```

## Expected Output
```
{'arguments': ['$(inputs.a.c.d)'],
'baseCommand': 'echo',
'class': 'CommandLineTool',
'cwlVersion': 'v1.2.0-dev3',
'doc': 'Tool that does something',
'http://commonwl.org/cwltool#original_cwlVersion': 'v1.1',
'id': 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/nested_record.cwl',
'inputs': [ordereddict([('type', ordereddict([('label', 'A record schema'), ('type', 'record'), ('fields', [ordereddict([('doc', 'This is inputs.a.b'), ('label', 'Some integer'), ('type', 'int'), ('name', 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/nested_record.cwl#a/b')]), ordereddict([('label', 'This is inputs.a.c'), ('type', [ordereddict([('type', 'record'), ('label', 'A nested record schema'), ('fields', [ordereddict([('type', 'float'), ('name', 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/nested_record.cwl#a/c/d')])])])]), ('name', 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/nested_record.cwl#a/c')])])])), ('id', 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/nested_record.cwl#a')])],
'label': 'Something tool',
'outputs': []}
DEBUG [job nested_record.cwl] initializing from file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/nested_record.cwl
DEBUG [job nested_record.cwl] {
"a": {
"b": 1,
"c": {
"d": 1.0
}
}
}
DEBUG [job nested_record.cwl] path mappings is {}
DEBUG [job nested_record.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "echo"
},
{
"position": [
0,
0
],
"valueFrom": "$(inputs.a.c.d)"
}
]
DEBUG [job nested_record.cwl] initial work dir {}
INFO [job nested_record.cwl] /tmp/6dbcyatw$ echo \
1.0
1.0
DEBUG Could not collect memory usage, job ended before monitoring began.
INFO [job nested_record.cwl] completed success
DEBUG [job nested_record.cwl] outputs {}
DEBUG [job nested_record.cwl] Removing input staging directory /tmp/tmptwsqgs98
DEBUG [job nested_record.cwl] Removing temporary directory /tmp/tmp3m9z6l9u
{}
INFO Final process status is success
```

## Your Environment
* cwltool version: 3.0.20200324120055

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.