common-workflow-language / common-workflow-language/cwltool
Input binding error for type union involving arrays and records
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Behavior
An input defined as a type union of an array schema that contains an item of type record schema causes cwltool to fail. The error may actually be due to an issue in schema_salad, but manifests itself during the `bind_input` in builder.py. The schema has already been processed by `make_avsc_object`, yet this is repeated when the type in the union does not have a name field found in the `Names` object. Yet, the record schema within the array already has been added, so it then fails `make_avsc_object` since the names already been added (see traceback).
By removing the avsc validation code avoids the error, and the process completes successfuly, since in fact the schema is valid.
Additionally, this piece of code appears to be insufficient for properly determining the input type at run-time for type unions in general, since it simply uses the first valid schema, regardless of what the input.
```
...
def bind_input(
...
# Handle union types
if isinstance(schema["type"], MutableSequence):
bound_input = False
for t in schema["type"]:
avsc = None # type: Optional[Schema]
if isinstance(t, str) and self.names.has_name(t, None):
avsc = self.names.get_name(t, None)
elif (
isinstance(t, MutableMapping)
and "name" in t
and self.names.has_name(cast(str, t["name"]), None)
):
avsc = self.names.get_name(cast(str, t["name"]), None)
if not avsc:
avsc = make_avsc_object(convert_to_dict(t), self.names)
if validate(avsc, datum):
...
```
## Workflow Code
```
cwlVersion : v1.1
class : CommandLineTool
label : Array union type error
doc : Error example
baseCommand : echo
arguments : ["hello"]
inputs :
an_array:
type:
- type: array
items:
- type: record
fields:
x : float
outputs : {}
requirements : {}
```
## Job Input
```
an_array:
- { x: 0.1 }
```
## Expected Output
```
{'arguments': ['hello'],
'baseCommand': 'echo',
'class': 'CommandLineTool',
'cwlVersion': 'v1.2.0-dev3',
'doc': 'Error example',
'http://commonwl.org/cwltool#original_cwlVersion': 'v1.1',
'id': 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/array_schema.cwl',
'inputs': [ordereddict([('type', [ordereddict([('type', 'array'), ('items', [ordereddict([('type', 'record'), ('fields', [ordereddict([('type', 'float'), ('name', 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/array_schema.cwl#an_array/x')])])])])])]), ('id', 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/array_schema.cwl#an_array')])],
'label': 'Array union type error',
'outputs': [],
'requirements': []}
DEBUG [job array_schema.cwl] initializing from file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/array_schema.cwl
DEBUG [job array_schema.cwl] {
"an_array": [
{
"x": 0.1
}
]
}
DEBUG [job array_schema.cwl] path mappings is {}
DEBUG [job array_schema.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "echo"
},
{
"position": [
0,
0
],
"datum": "hello"
}
]
DEBUG [job array_schema.cwl] initial work dir {}
INFO [job array_schema.cwl] /tmp/bzh045ng$ echo \
hello
hello
DEBUG Could not collect memory usage, job ended before monitoring began.
INFO [job array_schema.cwl] completed success
DEBUG [job array_schema.cwl] outputs {}
DEBUG [job array_schema.cwl] Removing input staging directory /tmp/tmp6dk7d65_
DEBUG [job array_schema.cwl] Removing temporary directory /tmp/tmp2_f471x0
{}
INFO Final process status is success
```
## Full Traceback (Actual)
```pytb
$ cwltool --debug array_schema.cwl array_job.yml
INFO /home/cdodd/.local/bin//cwltool 3.0.20200324120055
INFO Resolved 'array_schema.cwl' to 'file:///media/cdodd/box/projects/gembio-n1893/nanojig/examples/tools/array_schema.cwl'
ERROR Got workflow error
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 433, in __init__
new_schema = make_avsc_object(schema, names) # type: ignore
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 582, in make_avsc_object
name, namespace, fields, names, atype, doc, other_props
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 511, in __init__
NamedSchema.__init__(self, schema_type, name, namespace, names, other_props)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 247, in __init__
new_name = names.add_name(name, namespace, self)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 213, in add_name
raise SchemaParseException(fail_msg)
schema_salad.avro.schema.SchemaParseException: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 396, in __init__
items_schema = make_avsc_object(items, names)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 594, in make_avsc_object
return UnionSchema(json_data, names)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 436, in __init__
"Union item must be a valid Avro schema: %s" % str(err)
schema_salad.avro.schema.SchemaParseException: Union item must be a valid Avro schema: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/executors.py", line 221, in run_jobs
for job in jobiter:
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 551, in job
builder = self._init_job(job_order, runtimeContext)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/process.py", line 860, in _init_job
discover_secondaryFiles=getdefault(runtime_context.toplevel, False),
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/builder.py", line 336, in bind_input
discover_secondaryFiles=discover_secondaryFiles,
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/builder.py", line 267, in bind_input
avsc = make_avsc_object(convert_to_dict(t), self.names)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 588, in make_avsc_object
return ArraySchema(items, names, other_props)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 400, in __init__
"names: %s)" % (items, err, list(names.names.keys()))
schema_salad.avro.schema.SchemaParseException: Items schema ([{'type': 'record', 'fields': [{'type': 'float', 'name': 'x'}], 'name': 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2'}]) not a valid Avro schema: Union item must be a valid Avro schema: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use. (known names: ['File', 'File_class', 'Directory', 'Directory_class', 'Any', 'input_record_schema', 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2', 'outputs_record_schema'])
ERROR Workflow error:
Items schema ([{'type': 'record', 'fields': [{'type': 'float', 'name': 'x'}], 'name': 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2'}]) not a valid Avro schema: Union item must be a valid Avro schema: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use. (known names: ['File', 'File_class', 'Directory', 'Directory_class', 'Any', 'input_record_schema', 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2', 'outputs_record_schema'])
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 433, in __init__
new_schema = make_avsc_object(schema, names) # type: ignore
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 582, in make_avsc_object
name, namespace, fields, names, atype, doc, other_props
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 511, in __init__
NamedSchema.__init__(self, schema_type, name, namespace, names, other_props)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 247, in __init__
new_name = names.add_name(name, namespace, self)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 213, in add_name
raise SchemaParseException(fail_msg)
schema_salad.avro.schema.SchemaParseException: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 396, in __init__
items_schema = make_avsc_object(items, names)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 594, in make_avsc_object
return UnionSchema(json_data, names)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 436, in __init__
"Union item must be a valid Avro schema: %s" % str(err)
schema_salad.avro.schema.SchemaParseException: Union item must be a valid Avro schema: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/executors.py", line 221, in run_jobs
for job in jobiter:
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 551, in job
builder = self._init_job(job_order, runtimeContext)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/process.py", line 860, in _init_job
discover_secondaryFiles=getdefault(runtime_context.toplevel, False),
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/builder.py", line 336, in bind_input
discover_secondaryFiles=discover_secondaryFiles,
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/builder.py", line 267, in bind_input
avsc = make_avsc_object(convert_to_dict(t), self.names)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 588, in make_avsc_object
return ArraySchema(items, names, other_props)
File "/home/cdodd/.local/lib/python3.6/site-packages/schema_salad/avro/schema.py", line 400, in __init__
"names: %s)" % (items, err, list(names.names.keys()))
schema_salad.avro.schema.SchemaParseException: Items schema ([{'type': 'record', 'fields': [{'type': 'float', 'name': 'x'}], 'name': 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2'}]) not a valid Avro schema: Union item must be a valid Avro schema: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use. (known names: ['File', 'File_class', 'Directory', 'Directory_class', 'Any', 'input_record_schema', 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2', 'outputs_record_schema'])
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/main.py", line 1105, in main
tool, initialized_job_order_object, runtimeContext, logger=_logger
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/executors.py", line 54, in __call__
return self.execute(*args, **kwargs)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/executors.py", line 137, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/home/cdodd/.local/lib/python3.6/site-packages/cwltool/executors.py", line 255, in run_jobs
raise WorkflowException(str(err)) from err
cwltool.errors.WorkflowException: Items schema ([{'type': 'record', 'fields': [{'type': 'float', 'name': 'x'}], 'name': 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2'}]) not a valid Avro schema: Union item must be a valid Avro schema: The name "an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2" is already in use. (known names: ['File', 'File_class', 'Directory', 'Directory_class', 'Any', 'input_record_schema', 'an_array806703b5-560b-45bd-8f1a-f4c7c6b799e2', 'outputs_record_schema'])
```
Contributor guide
Research direction
Start in cwltool's builder.py at bind_input, then inspect schema_salad's make_avsc_object and validate paths shown in the traceback. Reproduce the failure with the supplied array_schema.cwl and array_job.yml examples. Done means the valid array-of-records union input completes successfully without duplicate-name errors and union validation selects the appropriate schema for the input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100