common-workflow-language / common-workflow-language/cwltool

'--pack' option fails when '--validate' works fine

Open
#1,103 1 comment 4 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
A full cwl-schema without links to any other files should be generated.

## Actual Behavior
An error occurs: `RuntimeError: [Errno 2] No such file or directory:`.

`cwltool` with `--pack` option can't deal with relative links inside subworkflows, but `--validate` definitely can.
When I replaced all relative links in this workflow to absolute paths the error changed to: `I'm sorry, I couldn't load this CWL file.`.

## Workflow Code
The workflow is here: `https://github.com/genome/analysis-workflows.git`: `definitions/pipelines/exome.cwl`

## Full Traceback
```pytb
bio@bisa:~/REPOs$ git clone https://github.com/genome/analysis-workflows.git
...
bio@bisa:~/REPOs$ cd analysis-workflows/definitions/

bio@bisa:~/REPOs/analysis-workflows/definitions$ cwltool --validate pipelines/exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'pipelines/exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Workflow checker warning:
...
pipelines/exome.cwl is valid CWL.
```
```pytb
bio@bisa:~/REPOs/analysis-workflows/definitions$ cwltool --pack pipelines/exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'pipelines/exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Tool definition failed initialization:
[Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'

bio@bisa:~/REPOs/analysis-workflows/definitions$ cwltool --pack --debug pipelines/exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'pipelines/exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Tool definition failed initialization:
[Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 154, in fetch_text
with open(urllib.request.url2pathname(str(path)), encoding='utf-8') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 1017, in fetch
text = self.fetch_text(url)
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 159, in fetch_text
raise RuntimeError(Text(err))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 663, in main
stdout.write(print_pack(loadingContext.loader, processobj, uri, metadata))
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 459, in print_pack
packed = pack(document_loader, processobj, uri, metadata)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 138, in pack
find_run(processobj, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 40, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 132, in loadref
return document_loader.resolve_ref(uri, base_url=base)[0]
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/sourceline.py", line 249, in __exit__
raise self.makeError(six.text_type(exc_value))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/subworkflows/bam_to_bqsr.cwl'
```
```pytb
bio@bisa:~/REPOs/analysis-workflows/definitions$ cd pipelines/

bio@bisa:~/REPOs/analysis-workflows/definitions/pipelines$ cwltool --pack --debug exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
Tool definition failed initialization:
[Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 154, in fetch_text
with open(urllib.request.url2pathname(str(path)), encoding='utf-8') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 1017, in fetch
text = self.fetch_text(url)
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 159, in fetch_text
raise RuntimeError(Text(err))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 663, in main
stdout.write(print_pack(loadingContext.loader, processobj, uri, metadata))
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 459, in print_pack
packed = pack(document_loader, processobj, uri, metadata)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 138, in pack
find_run(processobj, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 40, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 47, in find_run
find_run(s, loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 45, in find_run
find_run(loadref(None, d["run"]), loadref, runs)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 132, in loadref
return document_loader.resolve_ref(uri, base_url=base)[0]
File "/usr/local/lib/python3.6/dist-packages/schema_salad/ref_resolver.py", line 619, in resolve_ref
doc = self.fetch(doc_url, inject_ids=(not mixin))
File "/usr/local/lib/python3.6/dist-packages/schema_salad/sourceline.py", line 249, in __exit__
raise self.makeError(six.text_type(exc_value))
RuntimeError: [Errno 2] No such file or directory: '/home/bio/REPOs/analysis-workflows/definitions/pipelines/align.cwl'
```

And after making all paths to subworkflows and tools absolute:
```pytb
bio@bisa:~/REPOs/analysis-workflows/definitions/pipelines$ cwltool --pack --debug exome.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'exome.cwl' to 'file:///home/bio/REPOs/analysis-workflows/definitions/pipelines/exome.cwl'
I'm sorry, I couldn't load this CWL file.
The error was:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 663, in main
stdout.write(print_pack(loadingContext.loader, processobj, uri, metadata))
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 459, in print_pack
packed = pack(document_loader, processobj, uri, metadata)
File "/usr/local/lib/python3.6/dist-packages/cwltool/pack.py", line 195, in pack
v = rewrite[r]
KeyError: '../subworkflows/align.cwl'
```

## Your Environment
* cwltool version: 1.0.20190228155703

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.