common-workflow-language / common-workflow-language/cwltool

Workflow error with HTTP files and --cachedir

Open
#1,828 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

Having files HTTP files as input, defining a cache directory with the --cachedir option results in a _Workflow error_.
Runs successfully without.

## Expected Behavior
Same behaviour as without --cachedir defined. (But utilizing the cache functionality)

## Actual Behavior
Workflow Error

## Workflow Code
```
cwltool --debug --cachedir ./CACHE concatenate.cwl --infiles http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz --outname cattest
```
**concatenate.cwl**
```
cwlVersion: v1.2
class: CommandLineTool
label: "Concatenate multiple files"

baseCommand: [cat]

stdout: $(inputs.outname)

hints:
DockerRequirement:
dockerPull: debian:buster

inputs:
infiles:
type: File[]
inputBinding:
position: 2
outname:
type: string

outputs:
output:
type: File
outputBinding:
glob: $(inputs.outname)

```

## Full Traceback
```pytb
cwltool --debug --cachedir ./CACHE concatenate.cwl --infiles http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz --outname cattest
INFO /home/bart/.local/bin//cwltool 3.1.20230325110543
INFO Resolved 'concatenate.cwl' to 'file:///home/bart/UNLOCK/git-dev/cwl/tools/bash/concatenate.cwl'
DEBUG Parsed job order from command line: {
"id": "concatenate.cwl",
"infiles": [
{
"class": "File",
"location": "http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz"
}
],
"outname": "cattest"
}
ERROR Got workflow error
Traceback (most recent call last):
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 217, in run_jobs
for job in jobiter:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/command_line_tool.py", line 835, in job
visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 211, in visit_class
op(rec)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/process.py", line 1348, in compute_checksums
with fs_access.open(location, "rb") as f:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/stdfsaccess.py", line 38, in open
return open(self._abs(fn), mode)
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
ERROR Workflow error:
[Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
Traceback (most recent call last):
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 217, in run_jobs
for job in jobiter:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/command_line_tool.py", line 835, in job
visit_class([cachebuilder.files, cachebuilder.bindings], ("File"), _checksum)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 216, in visit_class
visit_class(d, cls, op)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/utils.py", line 211, in visit_class
op(rec)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/process.py", line 1348, in compute_checksums
with fs_access.open(location, "rb") as f:
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/stdfsaccess.py", line 38, in open
return open(self._abs(fn), mode)
FileNotFoundError: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/main.py", line 1295, in main
(out, status) = real_executor(
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 61, in __call__
return self.execute(process, job_order_object, runtime_context, logger)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 144, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/home/bart/.local/lib/python3.10/site-packages/cwltool/executors.py", line 251, in run_jobs
raise WorkflowException(str(err)) from err
cwltool.errors.WorkflowException: [Errno 2] No such file or directory: 'http://download.systemsbiology.nl/unlock/cwl/test_data/human_small.fa.gz'
```

## Your Environment
* cwltool version: 3.1.20230325110543

Contributor guide

Open the contributing guide

Research direction

Reproduce the command in the issue with --cachedir, then start at cwltool/command_line_tool.py around checksum computation and cwltool/stdfsaccess.py around file opening. Trace how the HTTP File location is handled with and without caching; done means the workflow completes with --cachedir and the remote input is cached and usable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.