common-workflow-language / common-workflow-language/cwltool

test_wf_without_container fails if the folder cache already exists

Open
#680 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I was running the tests locally and `test_wf_without_container` fails, which runs: `cwltool --cachedir cache tests/wf/hello-workflow.cwl --usermessage hello`. This was failing until I used the cachedir of `cache2` (I realised that I already had a existing folder called `cache`) - this behaviour indicates a problem with the caching logic. I've put a stack trace in debug mode below:

```
$ python -m cwltool --debug --cachedir cache tests/wf/hello-workflow.cwl --usermessage hello
/Users/th10/checkouts/cwltool/cwltool/__main__.py 1.0.20180209142616
Resolved 'tests/wf/hello-workflow.cwl' to 'file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl'
Parsed job order from command line: {
"id": "tests/wf/hello-workflow.cwl",
"usermessage": "hello"
}
[workflow hello-workflow.cwl] initialized from file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl
[workflow hello-workflow.cwl] start
[workflow hello-workflow.cwl] starting step step0
[job step step0] job input {
"file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl#step0/message": "hello"
}
[job step step0] evaluated job input to {
"file:///Users/th10/checkouts/cwltool/tests/wf/hello-workflow.cwl#step0/message": "hello"
}
[step step0] start
[job step0] keydictstr is {"cmdline":["echo","-n","-e","hello"],"stdout":"response.txt"} -> a7828ac93d7bbe212cf35e772e4f999b
[job step0] Using cached output in /Users/th10/checkouts/cwltool/cache/a7828ac93d7bbe212cf35e772e4f999b
Workflow error, try again with --debug for more information:
Error collecting output for parameter 'response':
tests/wf/hello-workflow.cwl:34:9: Traceback (most recent call last):

File
"/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py",
line 653, in collect_output
raise WorkflowException("Did not find output file with glob
pattern: '{}'".format(globpatterns))

cwltool.errors.WorkflowException: Did not find output file with
glob pattern: '['response.txt']'

During handling of the above exception, another exception
occurred:

Traceback (most recent call last):

File
"/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py",
line 538, in collect_output_ports
compute_checksum=compute_checksum)

File
"/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py",
line 653, in collect_output
raise WorkflowException("Did not find output file with glob
pattern: '{}'".format(globpatterns))

File
"/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py",
line 155, in __exit__
raise
self.makeError("\n".join(traceback.format_exception(exc_type,
exc_value, tb)))

cwltool.errors.WorkflowException: test:1:1: Traceback (most recent call last):
test:1:1:
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in
collect_output
raise WorkflowException("Did not find output file with glob pattern:
'{}'".format(globpatterns))

cwltool.errors.WorkflowException: Did not find output file with glob pattern:
'['response.txt']'
Traceback (most recent call last):
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 538, in collect_output_ports
compute_checksum=compute_checksum)
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 155, in __exit__
raise self.makeError("\n".join(traceback.format_exception(exc_type, exc_value, tb)))
cwltool.errors.WorkflowException: test:1:1: Traceback (most recent call last):
test:1:1:
test:1:1: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
test:1:1: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
test:1:1:
test:1:1: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/th10/checkouts/cwltool/cwltool/main.py", line 588, in main
**vars(args))
File "/Users/th10/checkouts/cwltool/cwltool/executors.py", line 29, in __call__
return self.execute(*args, **kwargs)
File "/Users/th10/checkouts/cwltool/cwltool/executors.py", line 69, in execute
self.run_jobs(t, job_order_object, logger, **kwargs)
File "/Users/th10/checkouts/cwltool/cwltool/executors.py", line 105, in run_jobs
r.run(**kwargs)
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 173, in run
kwargs.get("compute_checksum", True)), "success")
File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 538, in collect_output_ports
compute_checksum=compute_checksum)
File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 155, in __exit__
raise self.makeError("\n".join(traceback.format_exception(exc_type, exc_value, tb)))
cwltool.errors.WorkflowException: Error collecting output for parameter 'response':
tests/wf/hello-workflow.cwl:34:9: Traceback (most recent call last):
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
tests/wf/hello-workflow.cwl:34:9: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: During handling of the above exception, another exception occurred:
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: Traceback (most recent call last):
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 538, in collect_output_ports
tests/wf/hello-workflow.cwl:34:9: compute_checksum=compute_checksum)
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
tests/wf/hello-workflow.cwl:34:9: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
tests/wf/hello-workflow.cwl:34:9:
tests/wf/hello-workflow.cwl:34:9: File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 155, in __exit__
tests/wf/hello-workflow.cwl:34:9: raise self.makeError("\n".join(traceback.format_exception(exc_type, exc_value, tb)))
tests/wf/hello-workflow.cwl:34:9:
cwltool.errors.WorkflowException: test:1:1: Traceback (most recent call last):
test:1:1:
test:1:1: File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 653, in collect_output
test:1:1: raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
test:1:1:
test:1:1: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['response.txt']'
```

cwltool version: 1.0.20180209142616

Contributor guide

Open the contributing guide

Research direction

Start by running the reported test command, `cwltool --cachedir cache tests/wf/hello-workflow.cwl --usermessage hello`, with an existing cache directory. Inspect the cache-hit path and output collection in `cwltool/command_line_tool.py`, especially lines 538 and 653. Done means `test_wf_without_container` passes when the cache directory already exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.