common-workflow-language / common-workflow-language/cwltool
test_wf_without_container fails if the folder cache already exists
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
I was running the tests locally and `test_wf_without_container` fails, which runs: `cwltool --cachedir cache tests/wf/hello-workflow.cwl --usermessage hello`. This was failing until I used the cachedir of `cache2` (I realised that I already had a existing folder called `cache`) - this behaviour indicates a problem with the caching logic. I've put a stack trace in debug mode below:
```
$ python -m cwltool --debug --cachedir cache tests/wf/hello-workflow.cwl --usermessage hello
/Users/th10/checkouts/cwltool/cwltool/__main__.py 1.0.20180209142616
Resolved 'tests/wf/hello-workflow.cwl' to 'file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl'
Parsed job order from command line: {
"id": "tests/wf/hello-workflow.cwl",
"usermessage": "hello"
}
[workflow hello-workflow.cwl] initialized from file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl
[workflow hello-workflow.cwl] start
[workflow hello-workflow.cwl] starting step step0
[job step step0] job input {
"file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl#step0/message": "hello"
}
[job step step0] evaluated job input to {
"file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl#step0/message": "hello"
}
[step step0] start
[job step0] keydictstr is {"cmdline":["echo","-n","-e","hello"],"stdout":"response.txt"} -> a7828ac93d7bbe212cf35e772e4f999b
[job step0] Using cached output in /Users/th10/checkouts/cwltool/cache/a7828ac93d7bbe212cf35e772e4f999b
Workflow error, try again with --debug for more information:
Error collecting output for parameter 'response':
tests/wf/hello-workflow.cwl:34:9: Traceback (most recent call last):
File
"/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py",
line 653, in collect_output
raise WorkflowException("Did not find output file with glob
pattern: '{}'".format(globpatterns))
cwltool.errors.WorkflowException: Did not find output file with
glob pattern: '['response.txt']'
During handling of the above exception, another exception
occurred:
Traceback (most recent call last):
File
"/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py",
line 538, in collect_output_ports
compute_checksum=compute_checksum)
File
"/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py",
line 653, in collect_output
raise WorkflowException("Did not find output file with glob
pattern: '{}'".format(globpatterns))
File
"/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py",
line 155, in __exit__
raise
self.makeError("\n".join(traceback.format_exception(exc_type,
exc_value, tb)))
cwltool.errors.WorkflowException: test:1:1: Traceback (most recent call last):
test:1:1:
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in
collect_output
raise WorkflowException("Did not find output file with glob pattern:
'{}'".format(globpatterns))
cwltool.errors.WorkflowException: Did not find output file with glob pattern:
'['response.txt']'
Traceback (most recent call last):
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 538, in collect_output_ports
compute_checksum=compute_checksum)
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 155, in __exit__
raise self.makeError("\n".join(traceback.format_exception(exc_type, exc_value, tb)))
cwltool.errors.WorkflowException: test:1:1: Traceback (most recent call last):
test:1:1:
test:1:1: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
test:1:1: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
test:1:1:
test:1:1: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/th10/checkouts/cwltool/cwltool/main.py", line 588, in main
**vars(args))
File "/Users/th10/checkouts/cwltool/cwltool/executors.py", line 29, in __call__
return self.execute(*args, **kwargs)
File "/Users/th10/checkouts/cwltool/cwltool/executors.py", line 69, in execute
self.run_jobs(t, job_order_object, logger, **kwargs)
File "/Users/th10/checkouts/cwltool/cwltool/executors.py", line 105, in run_jobs
r.run(**kwargs)
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 173, in run
kwargs.get("compute_checksum", True)), "success")
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 538, in collect_output_ports
compute_checksum=compute_checksum)
File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 155, in __exit__
raise self.makeError("\n".join(traceback.format_exception(exc_type, exc_value, tb)))
cwltool.errors.WorkflowException: Error collecting output for parameter 'response':
tests/wf/hello-workflow.cwl:34:9: Traceback (most recent call last):
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
tests/wf/hello-workflow.cwl:34:9: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: During handling of the above exception, another exception occurred:
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: Traceback (most recent call last):
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 538, in collect_output_ports
tests/wf/hello-workflow.cwl:34:9: compute_checksum=compute_checksum)
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
tests/wf/hello-workflow.cwl:34:9: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 155, in __exit__
tests/wf/hello-workflow.cwl:34:9: raise self.makeError("\n".join(traceback.format_exception(exc_type, exc_value, tb)))
tests/wf/hello-workflow.cwl:34:9:
cwltool.errors.WorkflowException: test:1:1: Traceback (most recent call last):
test:1:1:
test:1:1: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
test:1:1: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
test:1:1:
test:1:1: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'
```
cwltool version: 1.0.20180209142616
Contributor guide
Research direction
Start by running the reported test command, `cwltool --cachedir cache tests/wf/hello-workflow.cwl --usermessage hello`, with an existing cache directory. Inspect the cache-hit path and output collection in `cwltool/command_line_tool.py`, especially lines 538 and 653. Done means `test_wf_without_container` passes when the cache directory already exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100