common-workflow-language / common-workflow-language/cwltool

location url not executed correctly with --cachedir CACHE

Open
#1,842 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior

When executing a yaml file with `cwltool --cachedir CACHE WORKFLOW.cwl` the input file with

```
cwl:tool: ../../workflows/workflow_ngtax.cwl
for_read_len: 100
forward_primer: '[AG]GGATTAGATACCC'
forward_reads:
- class: File
location: http://download.systemsbiology.nl/unlock/cwl/test_data/amplicon/forward.fastq.gz
memory: 6000
minimum_threshold: 0.1
reference_db:
class: File
location: /unlock/references/databases/Silva/SILVA_138.1_SSURef_tax_silva.fasta.gz
rev_read_len: 100
fragment: V3
reverse_primer: CGAC[AG][AG]CCATGCA[ACGT]CACCT
reverse_reads:
class: File
location: http://download.systemsbiology.nl/unlock/cwl/test_data/amplicon/reverse.fastq.gz
sample: UNLOCK_NGTAX_TEST
primersRemoved: true
```

Results in an
```
INFO [workflow ] start
INFO [workflow ] starting step prepare_fasta_db
INFO [step prepare_fasta_db] start
INFO [workflow prepare_fasta_db] start
INFO [workflow prepare_fasta_db] starting step prepare_fasta_db_2
INFO [step prepare_fasta_db_2] start
ERROR Unexpected exception
Traceback (most recent call last):
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/workflow.py", line 459, in job
yield from self.embedded_tool.job(
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/command_line_tool.py", line 835, in job
visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/utils.py", line 211, in visit_class
op(rec)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/process.py", line 1348, in compute_checksums
with fs_access.open(location, "rb") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/stdfsaccess.py", line 38, in open
return open(self._abs(fn), mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
ERROR [step prepare_fasta_db_2] Cannot make job: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
INFO [workflow prepare_fasta_db] completed permanentFail
WARNING [step prepare_fasta_db] completed permanentFail
INFO [workflow ] completed permanentFail
{
"filtered_reads": null,
"filtlong_log": null,
"kraken2_folder": null,
"nanoplot_filtered_folder": null,
"nanoplot_unfiltered_folder": null,
"reference_filter_longreads_log": null
}WARNING Final process status is permanentFail
```

## Actual Behavior

To be able to download the file and execute the workflow

## Workflow Code
```

https://workflowhub.eu/workflows/45

```

## Full Traceback
```pytb
INFO [workflow ] start
INFO [workflow ] starting step prepare_fasta_db
INFO [step prepare_fasta_db] start
INFO [workflow prepare_fasta_db] start
INFO [workflow prepare_fasta_db] starting step prepare_fasta_db_2
INFO [step prepare_fasta_db_2] start
ERROR Unexpected exception
Traceback (most recent call last):
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/workflow.py", line 459, in job
yield from self.embedded_tool.job(
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/command_line_tool.py", line 835, in job
visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/utils.py", line 211, in visit_class
op(rec)
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/process.py", line 1348, in compute_checksums
with fs_access.open(location, "rb") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/Git/m-unlock/cwl/venv/lib/python3.11/site-packages/cwltool/stdfsaccess.py", line 38, in open
return open(self._abs(fn), mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
ERROR [step prepare_fasta_db_2] Cannot make job: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
INFO [workflow prepare_fasta_db] completed permanentFail
WARNING [step prepare_fasta_db] completed permanentFail
INFO [workflow ] completed permanentFail
{
"filtered_reads": null,
"filtlong_log": null,
"kraken2_folder": null,
"nanoplot_filtered_folder": null,
"nanoplot_unfiltered_folder": null,
"reference_filter_longreads_log": null
}WARNING Final process status is permanentFail
```

## Your Environment
* cwltool version: 3.1.20230513155734

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.