common-workflow-language / common-workflow-language/cwltool
Unable to use the $import directive
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
Section [2.4 (Document preprocessing)](https://www.commonwl.org/v1.1/CommandLineTool.html#Document_preprocessing) (of the Common Workflow Language Command Line Tool Description v1.1) mentions the possible usage of the `$import` directive. One can find some [rare examples](https://github.com/EBI-Metagenomics/ebi-metagenomics-cwl/blob/master/workflows/emg-core-analysis-v4.cwl#L10) of usage of this `$import` directive that don't seem that straightforward to transpose (at least for me and when using `cwl-runner` version 1.0.20190831161204).
For example suppose we realized the following break down of the [first example of the user guide](https://www.commonwl.org/user_guide/02-1st-example/index.html) as the following two workflows
_to-be-imported.cwl_
```
inputs:
message:
type: string
inputBinding:
position: 1
outputs: []
```
and
_main.cwl_
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.1
class: CommandLineTool
baseCommand: echo
$import: to-be-imported.cwl
```
while keeping the `echo-job.yml` argument file provided in the user guide. And further assume that you try to run this broken down first example with the following bash commands
```
virtualenv -p python3 venv
source venv/bin/activate
pip install cwlref-runner
cwl-runner main.cwl echo-job.yml
```
The expected behavior is thus that the import directive does indeed import the designated piece of workflow.
## Actual Behavior
Instead, when running the above mentioned command `cwl-runner main.cwl echo-job.yml
` then you might probably get the following error message
```
ERROR Tool definition failed initialization:
main.cwl:6:1: '$import' must be the only field in ordereddict([('cwlVersion', 'v1.1'), ('class', 'CommandLineTool'), ('baseCommand', 'echo'), ('$import', 'to-be-imported.cwl'), ('id', 'file:///Users/eboix/SAVED/Divers/Ruses/CWL_examples/main.cwl')])
```
Notes:
* trying to use the line
```
{$import: to-be-imported.cwl}
```
in place of the line
```
$import: to-be-imported.cwl
```
as suggested by the ["Workflow steps and wiring" section of this issue](https://github.com/common-workflow-language/common-workflow-language/issues/173) doesn't seem to help.
* trying the following syntax
```
"@import": to-be-imported.cwl
```
as suggested by [this issue](https://github.com/common-workflow-language/cwltool/issues/26) also fails but with the following different error message
```
main.cwl:3:1: "outputs" section is not valid.
```
## Workflow Code
The following [import_problem.zip zip file](https://github.com/common-workflow-language/cwltool/files/3620135/import_problem.zip) provides all the code: start with the extracted `Readme.md` and refer to the above description.
## Full Traceback
```(venv) cwl-runner --debug main.cwl echo-job.yml
INFO /private/tmp/import_problem/venv/bin/cwl-runner 1.0.20190915164430
INFO Resolved 'main.cwl' to 'file:///private/tmp/import_problem/main.cwl'
ERROR I'm sorry, I couldn't load this CWL file.
The error was:
Traceback (most recent call last):
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/cwltool/main.py", line 661, in main
uri, loadingContext)
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/cwltool/load_tool.py", line 113, in fetch_document
workflowobj = loadingContext.loader.fetch(fileuri)
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/schema_salad/ref_resolver.py", line 1192, in fetch
result = next(attachments)
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/main.py", line 959, in load_all
while loader._constructor.check_data():
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 98, in check_data
return self.composer.check_node()
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/composer.py", line 58, in check_node
if self.parser.check_event(StreamStartEvent):
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/parser.py", line 141, in check_event
self.current_event = self.state()
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/parser.py", line 177, in parse_stream_start
token.move_comment(self.scanner.peek_token())
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/scanner.py", line 1775, in peek_token
self._gather_comments()
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/scanner.py", line 1803, in _gather_comments
self.fetch_more_tokens()
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/scanner.py", line 282, in fetch_more_tokens
return self.fetch_value()
File "/private/tmp/import_problem/venv/lib/python3.7/site-packages/ruamel/yaml/scanner.py", line 655, in fetch_value
self.reader.get_mark(),
ruamel.yaml.scanner.ScannerError: mapping values are not allowed here
in "file:///private/tmp/import_problem/main.cwl", line 4, column 11
```
## Your Environment
```
(venv): cwltool --version
/private/tmp/import_problem/venv/bin/cwltool 1.0.20190915164430
(venv) : python --version
Python 3.7.4
```
Contributor guide
Assessment
This issue has not been assessed yet.