common-workflow-language / common-workflow-language/cwltool
cachedir does not cache correctly when parameters are not used on the command line
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
if I use a CommandLineTool that uses the input parameters for a file built using `InitialWorkDirRequirement`, if I execute twice a tool in the same workflow but with different parameters, the cache results from the first execution should not be used for the second
## Actual Behavior
Instead of running twice the tool, `cwltool` will reuse the first cached run.
## Workflow Code
Test tool used:
```yaml
#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
baseCommand: ["ls", "in"]
requirements:
InitialWorkDirRequirement:
listing:
- entryname: "in"
entry: |
${
if(typeof inputs.para != 'undefined' && inputs.para != null){
return "para=" + inputs.para;
}else{
return "";
}
}
InlineJavascriptRequirement: {}
inputs:
para:
type: string?
name:
type: string
outputs:
test:
type: stdout
stdout: $(inputs.name)
```
Test workflow used:
```yaml
class: Workflow
cwlVersion: v1.0
id: initialworkdirrequirement_cwltool_cache_bug_demo_wf
label: InitialWorkDirRequirement cwltool cache bug demo WF
inputs: []
outputs:
- id: test
outputSource:
- test_iwdr_tool/test
type: File
- id: test_1
outputSource:
- test_iwdr_tool_1/test
type: File
steps:
- id: test_iwdr_tool
in:
- id: para
default: A
- id: name
default: fileA.txt
out:
- id: test
run: /home/hmenager/Desktop/test_iwdr_tool.cwl
- id: test_iwdr_tool_1
in:
- id: para
default: B
- id: name
default: fileB.txt
out:
- id: test
run: /home/hmenager/Desktop/test_iwdr_tool.cwl
```
## Full Traceback
Below you can find the result when running the workflow without and then with the cache.
```shell
$ cwltool initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl
/media/hmenager/datapasteur/11317_Role_of_ATM_in_EBV_latency/venv/bin/cwltool 1.0.20180923172926
Resolved 'initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl' to 'file:///home/hmenager/Desktop/initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl'
[workflow ] start
[workflow ] starting step test_iwdr_tool
[step test_iwdr_tool] start
[job test_iwdr_tool] /tmp/tmpMFfXOF$ ls \
in > /tmp/tmpMFfXOF/fileA.txt
[job test_iwdr_tool] completed success
[step test_iwdr_tool] completed success
[workflow ] starting step test_iwdr_tool_1
[step test_iwdr_tool_1] start
[job test_iwdr_tool_1] /tmp/tmp__iCBa$ ls \
in > /tmp/tmp__iCBa/fileB.txt
[job test_iwdr_tool_1] completed success
[step test_iwdr_tool_1] completed success
[workflow ] completed success
{
"test": {
"checksum": "sha1$9d26586a7869bfe07eec69d43beda236ad152297",
"basename": "fileA.txt",
"location": "file:///home/hmenager/Desktop/fileA.txt",
"path": "/home/hmenager/Desktop/fileA.txt",
"class": "File",
"size": 3
},
"test_1": {
"checksum": "sha1$9d26586a7869bfe07eec69d43beda236ad152297",
"basename": "fileB.txt",
"location": "file:///home/hmenager/Desktop/fileB.txt",
"path": "/home/hmenager/Desktop/fileB.txt",
"class": "File",
"size": 3
}
}
Final process status is success
$ cwltool --cachedir tutu initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl
/media/hmenager/datapasteur/11317_Role_of_ATM_in_EBV_latency/venv/bin/cwltool 1.0.20180923172926
Resolved 'initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl' to 'file:///home/hmenager/Desktop/initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl'
[workflow ] start
[workflow ] starting step test_iwdr_tool_1
[step test_iwdr_tool_1] start
[job test_iwdr_tool_1] Output of job will be cached in /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235
[job test_iwdr_tool_1] /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235$ ls \
in > /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235/fileB.txt
[job test_iwdr_tool_1] completed success
[step test_iwdr_tool_1] completed success
[workflow ] starting step test_iwdr_tool
[step test_iwdr_tool] start
[job test_iwdr_tool] Using cached output in /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235
Workflow error, try again with --debug for more information:
Error collecting output for parameter 'test':
test:1:1: Did not find output file with glob pattern: '['fileA.txt']
```
## Your Environment
* cwltool version:
cwltool 1.0.20180923172926
Contributor guide
Research direction
Start by reproducing the supplied CommandLineTool and workflow with cwltool --cachedir, comparing the two InitialWorkDirRequirement values and the cache behavior. Trace how cwltool identifies cached jobs and collects outputs; done means both executions use distinct results and the workflow completes successfully with fileA.txt and fileB.txt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100