common-workflow-language / common-workflow-language/cwltool

cachedir does not cache correctly when parameters are not used on the command line

Open
#962 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
if I use a CommandLineTool that uses the input parameters for a file built using `InitialWorkDirRequirement`, if I execute twice a tool in the same workflow but with different parameters, the cache results from the first execution should not be used for the second

## Actual Behavior
Instead of running twice the tool, `cwltool` will reuse the first cached run.
## Workflow Code
Test tool used:
```yaml
#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
baseCommand: ["ls", "in"]
requirements:
InitialWorkDirRequirement:
listing:
- entryname: "in"
entry: |
${
if(typeof inputs.para != 'undefined' && inputs.para != null){
return "para=" + inputs.para;
}else{
return "";
}
}
InlineJavascriptRequirement: {}
inputs:
para:
type: string?
name:
type: string
outputs:
test:
type: stdout
stdout: $(inputs.name)
```

Test workflow used:
```yaml
class: Workflow
cwlVersion: v1.0
id: initialworkdirrequirement_cwltool_cache_bug_demo_wf
label: InitialWorkDirRequirement cwltool cache bug demo WF
inputs: []
outputs:
- id: test
outputSource:
- test_iwdr_tool/test
type: File
- id: test_1
outputSource:
- test_iwdr_tool_1/test
type: File
steps:
- id: test_iwdr_tool
in:
- id: para
default: A
- id: name
default: fileA.txt
out:
- id: test
run: /home/hmenager/Desktop/test_iwdr_tool.cwl
- id: test_iwdr_tool_1
in:
- id: para
default: B
- id: name
default: fileB.txt
out:
- id: test
run: /home/hmenager/Desktop/test_iwdr_tool.cwl
```
## Full Traceback

Below you can find the result when running the workflow without and then with the cache.

```shell
$ cwltool initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl
/media/hmenager/datapasteur/11317_Role_of_ATM_in_EBV_latency/venv/bin/cwltool 1.0.20180923172926
Resolved 'initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl' to 'file:///home/hmenager/Desktop/initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl'
[workflow ] start
[workflow ] starting step test_iwdr_tool
[step test_iwdr_tool] start
[job test_iwdr_tool] /tmp/tmpMFfXOF$ ls \
in > /tmp/tmpMFfXOF/fileA.txt
[job test_iwdr_tool] completed success
[step test_iwdr_tool] completed success
[workflow ] starting step test_iwdr_tool_1
[step test_iwdr_tool_1] start
[job test_iwdr_tool_1] /tmp/tmp__iCBa$ ls \
in > /tmp/tmp__iCBa/fileB.txt
[job test_iwdr_tool_1] completed success
[step test_iwdr_tool_1] completed success
[workflow ] completed success
{
"test": {
"checksum": "sha1$9d26586a7869bfe07eec69d43beda236ad152297",
"basename": "fileA.txt",
"location": "file:///home/hmenager/Desktop/fileA.txt",
"path": "/home/hmenager/Desktop/fileA.txt",
"class": "File",
"size": 3
},
"test_1": {
"checksum": "sha1$9d26586a7869bfe07eec69d43beda236ad152297",
"basename": "fileB.txt",
"location": "file:///home/hmenager/Desktop/fileB.txt",
"path": "/home/hmenager/Desktop/fileB.txt",
"class": "File",
"size": 3
}
}
Final process status is success
$ cwltool --cachedir tutu initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl
/media/hmenager/datapasteur/11317_Role_of_ATM_in_EBV_latency/venv/bin/cwltool 1.0.20180923172926
Resolved 'initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl' to 'file:///home/hmenager/Desktop/initialworkdirrequirement-cwltool-cache-bug-demo-wf.cwl'
[workflow ] start
[workflow ] starting step test_iwdr_tool_1
[step test_iwdr_tool_1] start
[job test_iwdr_tool_1] Output of job will be cached in /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235
[job test_iwdr_tool_1] /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235$ ls \
in > /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235/fileB.txt
[job test_iwdr_tool_1] completed success
[step test_iwdr_tool_1] completed success
[workflow ] starting step test_iwdr_tool
[step test_iwdr_tool] start
[job test_iwdr_tool] Using cached output in /home/hmenager/Desktop/tutu/787fcb2d3bace4f0c05c3a04b19fa235
Workflow error, try again with --debug for more information:
Error collecting output for parameter 'test':
test:1:1: Did not find output file with glob pattern: '['fileA.txt']
```

## Your Environment
* cwltool version:
cwltool 1.0.20180923172926

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied CommandLineTool and workflow with cwltool --cachedir, comparing the two InitialWorkDirRequirement values and the cache behavior. Trace how cwltool identifies cached jobs and collects outputs; done means both executions use distinct results and the workflow completes successfully with fileA.txt and fileB.txt.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.