common-workflow-language / common-workflow-language/cwltool
Workflow error with HTTP files and --cachedir
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
Having files HTTP files as input, defining a cache directory with the --cachedir option results in a _Workflow error_.
Runs successfully without.
## Expected Behavior
Same behaviour as without --cachedir defined. (But utilizing the cache functionality)
## Actual Behavior
Workflow Error
## Workflow Code
```
cwltool --debug --cachedir ./CACHE concatenate.cwl --infiles http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz --outname cattest
```
**concatenate.cwl**
```
cwlVersion: v1.2
class: CommandLineTool
label: "Concatenate multiple files"
baseCommand: [cat]
stdout: $(inputs.outname)
hints:
DockerRequirement:
dockerPull: debian:buster
inputs:
infiles:
type: File[]
inputBinding:
position: 2
outname:
type: string
outputs:
output:
type: File
outputBinding:
glob: $(inputs.outname)
```
## Full Traceback
```pytb
cwltool --debug --cachedir ./CACHE concatenate.cwl --infiles http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz --outname cattest
INFO /home/bart/.local/bin//cwltool 3.1.20230325110543
INFO Resolved 'concatenate.cwl' to 'file:///home/bart/UNLOCK/git-dev/cwl/tools/bash/concatenate.cwl'
DEBUG Parsed job order from command line: {
"id": "concatenate.cwl",
"infiles": [
{
"class": "File",
"location": "http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz"
}
],
"outname": "cattest"
}
ERROR Got workflow error
Traceback (most recent call last):
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 217, in run_jobs
for job in jobiter:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/command_line_tool.py", line 835, in job
visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 211, in visit_class
op(rec)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/process.py", line 1348, in compute_checksums
with fs_access.open(location, "rb") as f:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/stdfsaccess.py", line 38, in open
return open(self._abs(fn), mode)
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
ERROR Workflow error:
[Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
Traceback (most recent call last):
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 217, in run_jobs
for job in jobiter:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/command_line_tool.py", line 835, in job
visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 211, in visit_class
op(rec)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/process.py", line 1348, in compute_checksums
with fs_access.open(location, "rb") as f:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/stdfsaccess.py", line 38, in open
return open(self._abs(fn), mode)
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/main.py", line 1295, in main
(out, status) = real_executor(
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 61, in __call__
return self.execute(process, job_order_object, runtime_context, logger)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 144, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 251, in run_jobs
raise WorkflowException(str(err)) from err
cwltool.errors.WorkflowException: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
```
## Your Environment
* cwltool version: 3.1.20230325110543
Contributor guide
Research direction
Reproduce the command in the issue with --cachedir, then start at cwltool/command_line_tool.py around checksum computation and cwltool/stdfsaccess.py around file opening. Trace how the HTTP File location is handled with and without caching; done means the workflow completes with --cachedir and the remote input is cached and usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100