common-workflow-language / common-workflow-language/cwltool
Behavior of `default` field of file and directory in CommandLineTool is strange
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
When I tried to set default path to File and Directory in CommandLineTool, cwltool behaved strangely.
## Actual Behavior
I created three CommandLineTools and checked their behavior when run the `--validate` and `--make-template`.
### good_command_line_tool
```cwl
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: ps
arguments: []
inputs:
input_string:
type: string
default: string_example
input_file:
type: File
default:
class: File
location: /path/to/file
input_dir:
type: Directory
default:
class: Directory
location: /path/to/dir
outputs: {}
```
Behavior
```shell
$ cwltool --validate good_command_line_tool.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'good_command_line_tool.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/good_command_line_tool.cwl'
good_command_line_tool.cwl:19:7: Field `location` contains undefined reference to `file:///path/to/dir`
good_command_line_tool.cwl:14:7: Field `location` contains undefined reference to `file:///path/to/file`
good_command_line_tool.cwl is valid CWL.
$ cwltool --make-template good_command_line_tool.cwl > good_command_line_tool_job.yml
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'good_command_line_tool.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/good_command_line_tool.cwl'
good_command_line_tool.cwl:19:7: Field `location` contains undefined reference to `file:///path/to/dir`
good_command_line_tool.cwl:14:7: Field `location` contains undefined reference to `file:///path/to/file`
$ cat good_command_line_tool_job.yml
input_string: string_example # default value of type "string".
input_file: # default value of type "File".
class: File
location: file:///path/to/file
input_dir: # default value of type "Directory".
class: Directory
location: file:///path/to/dir
$ cwltool good_command_line_tool.cwl good_command_line_tool_job.yml
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'good_command_line_tool.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/good_command_line_tool.cwl'
good_command_line_tool.cwl:19:7: Field `location` contains undefined reference to `file:///path/to/dir`
good_command_line_tool.cwl:14:7: Field `location` contains undefined reference to `file:///path/to/file`
Got workflow error
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/cwltool/executors.py", line 162, in run_jobs
for job in jobiter:
File "/usr/local/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 391, in job
builder = self._init_job(job_order, runtimeContext)
File "/usr/local/lib/python3.7/site-packages/cwltool/process.py", line 616, in _init_job
get_listing(fs_access, job, recursive=(load_listing == "deep_listing"))
File "/usr/local/lib/python3.7/site-packages/cwltool/pathmapper.py", line 127, in get_listing
get_listing(fs_access, f, recursive=recursive)
File "/usr/local/lib/python3.7/site-packages/cwltool/pathmapper.py", line 133, in get_listing
for ld in fs_access.listdir(loc):
File "/usr/local/lib/python3.7/site-packages/cwltool/stdfsaccess.py", line 56, in listdir
return [abspath(urllib.parse.quote(str(l)), fn) for l in os.listdir(self._abs(fn))]
FileNotFoundError: [Errno 2] No such file or directory: '/path/to/dir'
Workflow error, try again with --debug for more information:
[Errno 2] No such file or directory: '/path/to/dir'
```
It works as above, but a warning message like below is displayed.
```shell
good_command_line_tool.cwl:19:7: Field `location` contains undefined reference to `file:///path/to/dir`
good_command_line_tool.cwl:14:7: Field `location` contains undefined reference to `file:///path/to/file`
```
### bad_command_line_tool_1
```cwl
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: ps
arguments: []
inputs:
input_string:
type: string
default: string_example
input_file:
type: File
default: /path/to/file
input_dir:
type: Directory
default: /path/to/dir
outputs: {}
```
Behavior
```shell
$ cwltool --validate bad_command_line_tool_1.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'bad_command_line_tool_1.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/bad_command_line_tool_1.cwl'
bad_command_line_tool_1.cwl is valid CWL.
$ cwltool --make-template bad_command_line_tool_1.cwl > bad_command_line_tool_1_job.yml
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'bad_command_line_tool_1.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/bad_command_line_tool_1.cwl'
$ cat bad_command_line_tool_1_job.yml
input_string: string_example # default value of type "string".
input_file: /path/to/file # default value of type "File".
input_dir: /path/to/dir # default value of type "Directory".
$ cwltool bad_command_line_tool_1.cwl bad_command_line_tool_1_job.yml
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'bad_command_line_tool_1.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/bad_command_line_tool_1.cwl'
Workflow error, try again with --debug for more information:
Invalid job input record:
bad_command_line_tool_1_job.yml:3:1: * the `input_dir` field is not valid because
value is a str, expected null or Directory
bad_command_line_tool_1_job.yml:2:1: * the `input_file` field is not valid because
value is a str, expected null or File
```
The problems are:
- Passing `--validate`
- When `--make-template` is executed, File and Directory are evaluated as string.
### bad_command_line_tool_2
```cwl
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: ps
arguments: []
inputs:
input_string:
type: string
default: string_example
input_file:
type: File
default:
class: File
default: /path/to/file
input_dir:
type: Directory
default:
class: Directory
default: /path/to/dir
outputs: {}
```
Behavior
```shell
$ cwltool --validate bad_command_line_tool_2.cwl
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'bad_command_line_tool_2.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/bad_command_line_tool_2.cwl'
bad_command_line_tool_2.cwl is valid CWL.
$ cwltool --make-template bad_command_line_tool_2.cwl > bad_command_line_tool_2_job.yml
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'bad_command_line_tool_2.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/bad_command_line_tool_2.cwl'
$ cat bad_command_line_tool_2_job.yml
input_string: string_example # default value of type "string".
input_file: # default value of type "File".
class: File
default: /path/to/file
input_dir: # default value of type "Directory".
class: Directory
default: /path/to/dir
$ cwltool bad_command_line_tool_2.cwl bad_command_line_tool_2_job.yml
/usr/local/bin/cwltool 1.0.20190228155703
Resolved 'bad_command_line_tool_2.cwl' to 'file:///Users/suecharo/tmp/cwl_file_dir/bad_command_line_tool_2.cwl'
Traceback (most recent call last):
File "/usr/local/bin/cwltool", line 10, in
sys.exit(run())
File "/usr/local/lib/python3.7/site-packages/cwltool/main.py", line 906, in run
sys.exit(main(*args, **kwargs))
File "/usr/local/lib/python3.7/site-packages/cwltool/main.py", line 782, in main
secret_store=runtimeContext.secret_store)
File "/usr/local/lib/python3.7/site-packages/cwltool/main.py", line 370, in init_job_order
normalizeFilesDirs(job_order_object)
File "/usr/local/lib/python3.7/site-packages/cwltool/pathmapper.py", line 97, in normalizeFilesDirs
visit_class(job, ("File", "Directory"), addLocation)
File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 214, in visit_class
visit_class(rec[d], cls, op)
File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 212, in visit_class
op(rec)
File "/usr/local/lib/python3.7/site-packages/cwltool/pathmapper.py", line 61, in addLocation
raise validate.ValidationException("Anonymous file object must have 'contents' and 'basename' fields.")
schema_salad.validate.ValidationException: Anonymous file object must have 'contents' and 'basename' fields.
```
The problem is:
- There is no field `default` in the class specification of File or Directory.
## Your Environment
```shell
$ uname -a
Darwin suecharos-MBP.local 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
$ cwltool --version
/usr/local/bin/cwltool 1.0.20190228155703
```
Contributor guide
Assessment
This issue has not been assessed yet.