common-workflow-language / common-workflow-language/cwltool
'--pack' option fails when '--validate' works fine
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
A full cwl-schema without links to any other files should be generated.
## Actual Behavior
An error occurs: `RuntimeError: [Errno 2] No such file or directory:`.
`cwltool` with `--pack` option can't deal with relative links inside subworkflows, but `--validate` definitely can.
When I replaced all relative links in this workflow to absolute paths the error changed to: `I'm sorry, I couldn't load this CWL file.`.
## Workflow Code
The workflow is here: `https://github.com/genome/analysis-workflows.git`: `definitions/pipelines/exome.cwl`
## Full Traceback
```pytb
bio@bisa:~/REPOs$ git clone https://github.com/genome/analysis-workflows.git
...
bio@bisa:~/REPOs$ cd analysis-workflows/definitions/
bio@bisa:~/REPOs/analysis-workflows/definitions$ cwltool --validate pipelines/exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'pipelines/exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Workflow checker warning:
...
pipelines/exome.cwl is valid CWL.
```
```pytb
bio@bisa:~/REPOs/analysis-workflows/definitions$ cwltool --pack pipelines/exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'pipelines/exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Tool definition failed initialization:
[Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'
bio@bisa:~/REPOs/analysis-workflows/definitions$ cwltool --pack --debug pipelines/exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'pipelines/exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Tool definition failed initialization:
[Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 154, in fetch_text
with open(urllib.request.url2pathname(str(path)), encoding='utf-8') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 1017, in fetch
text = self.fetch_text(url)
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 159, in fetch_text
raise RuntimeError(Text(err))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 663, in main
stdout.write(print_pack(loadingContext.loader, processobj, uri, metadata))
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 459, in print_pack
packed = pack(document_loader, processobj, uri, metadata)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 138, in pack
find_run(processobj, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 40, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 132, in loadref
return document_loader.resolve_ref(uri, base_url=base)[0]
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/sourceline.py", line 249, in __exit__
raise self.makeError(six.text_type(exc_value))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'
```
```pytb
bio@bisa:~/REPOs/analysis-workflows/definitions$ cd pipelines/
bio@bisa:~/REPOs/analysis-workflows/definitions/pipelines$ cwltool --pack --debug exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Tool definition failed initialization:
[Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 154, in fetch_text
with open(urllib.request.url2pathname(str(path)), encoding='utf-8') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 1017, in fetch
text = self.fetch_text(url)
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 159, in fetch_text
raise RuntimeError(Text(err))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 663, in main
stdout.write(print_pack(loadingContext.loader, processobj, uri, metadata))
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 459, in print_pack
packed = pack(document_loader, processobj, uri, metadata)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 138, in pack
find_run(processobj, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 40, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 132, in loadref
return document_loader.resolve_ref(uri, base_url=base)[0]
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/sourceline.py", line 249, in __exit__
raise self.makeError(six.text_type(exc_value))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'
```
And after making all paths to subworkflows and tools absolute:
```pytb
bio@bisa:~/REPOs/analysis-workflows/definitions/pipelines$ cwltool --pack --debug exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
I'm sorry, I couldn't load this CWL file.
The error was:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 663, in main
stdout.write(print_pack(loadingContext.loader, processobj, uri, metadata))
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 459, in print_pack
packed = pack(document_loader, processobj, uri, metadata)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 195, in pack
v = rewrite[r]
KeyError: '../subworkflows/align.cwl'
```
## Your Environment
* cwltool version: 1.0.20190228155703
Contributor guide
Assessment
This issue has not been assessed yet.