common-workflow-language / common-workflow-language/cwltool
Invalid filename if it contains #
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
File `path`s should be recognized without URI escaping (Except whatever is needed by YAML) even if they contain URI-unfriendly characters like `#` or ` `; e.g. `colo-829_1#111.fq.gz`
Such paths should also work as command line arguments.
File `location` with URIs that contain correct escaping like `%23` should generate the correct `basename` without truncating.
File `basename` with `#` should be accepted.
## Actual Behavior
cwltool tries to look for the truncated file name `colo-829_1`
## Workflow Code
Reported [initially](https://gitter.im/common-workflow-language/common-workflow-language?at=5c8fafa3f3dbbd230c77c2b6) by @byb121
where filenames like `colo-829_1#111.fq.gz` are not recognized by `cwltool`.
**inp.cwl**
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: cat
inputs:
example_file:
type: File?
inputBinding:
position: 1
outputs:
example_out:
type: stdout
```
**inp-job.yml**
```
example_file:
class: File
path: whale#1.txt
```
## Full Traceback
Using the above **inp-job.yml** fails with not finding the file `whale` - truncating after `#` in the `basename` and showing `#1.txt` wrongly unescaped as an URI fragment in `location`.
```pytb
(cwldev) stain@biggie:~/src/cwltool/92$ cwltool --debug inp.cwl inp-job.yml
/home/stain/miniconda3/envs/cwldev/bin/cwltool 1.0.20190228200117
Resolved 'inp.cwl' to 'file:///home/stain/src/cwltool/92/inp.cwl'
[job inp.cwl] initializing from file:///home/stain/src/cwltool/92/inp.cwl
[job inp.cwl] {
"example_file": {
"class": "File",
"location": "file:///home/stain/src/cwltool/92/whale#1.txt",
"size": 0,
"basename": "whale",
"nameroot": "whale",
"nameext": ""
}
}
Got workflow error
Traceback (most recent call last):
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 162, in run_jobs
for job in jobiter:
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 422, in job
visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 212, in visit_class
op(rec)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 209, in check_adjust
builder.pathmapper.mapper(file_o["location"])[1])
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/pathmapper.py", line 323, in mapper
p = self._pathmap[src[:i]]
KeyError: 'file:///home/stain/src/cwltool/92/whale'
Workflow error:
'file:///home/stain/src/cwltool/92/whale'
Traceback (most recent call last):
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 162, in run_jobs
for job in jobiter:
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 422, in job
visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 212, in visit_class
op(rec)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 209, in check_adjust
builder.pathmapper.mapper(file_o["location"])[1])
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/pathmapper.py", line 323, in mapper
p = self._pathmap[src[:i]]
KeyError: 'file:///home/stain/src/cwltool/92/whale'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/main.py", line 813, in main
logger=_logger)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 39, in __call__
return self.execute(*args, **kwargs)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 101, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 190, in run_jobs
raise WorkflowException(Text(err))
cwltool.errors.WorkflowException: 'file:///home/stain/src/cwltool/92/whale'
```
### As command line argument
Providing instead on the command line argument also fails, this time with `Invalid filename`.
```pytb
(cwldev) stain@biggie:~/src/cwltool/92$ cwltool --debug inp.cwl --example_file whale#1.txt
/home/stain/miniconda3/envs/cwldev/bin/cwltool 1.0.20190228200117
Resolved 'inp.cwl' to 'file:///home/stain/src/cwltool/92/inp.cwl'
Parsed job order from command line: {
"id": "inp.cwl",
"example_file": {
"class": "File",
"location": "file:///home/stain/src/cwltool/92/whale%231.txt"
}
}
[job inp.cwl] initializing from file:///home/stain/src/cwltool/92/inp.cwl
[job inp.cwl] {
"example_file": {
"class": "File",
"location": "file:///home/stain/src/cwltool/92/whale%231.txt",
"size": 0,
"basename": "whale#1.txt",
"nameroot": "whale#1",
"nameext": ".txt"
}
}
Workflow error:
Invalid filename: 'whale#1.txt' contains illegal characters
Traceback (most recent call last):
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/main.py", line 813, in main
logger=_logger)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 39, in __call__
return self.execute(*args, **kwargs)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 101, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 162, in run_jobs
for job in jobiter:
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 422, in job
visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 212, in visit_class
op(rec)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 224, in check_adjust
file_o["basename"]))
cwltool.errors.WorkflowException: Invalid filename: 'whale#1.txt' contains illegal characters
```
## Workaround
A workaround is to use `location` instead of `path` and use `%` URI escaping so that `#` turns into `%23`. However you also need to provide a `basename` that does not include the `#`.
**inp-job-workaround.yml**
```
example_file:
class: File
location: whale%231.txt
basename: whale_1.txt
```
works:
```pytb
(cwldev) stain@biggie:~/src/cwltool/92$ cwltool --debug inp.cwl inp-job-workaround.yml
/home/stain/miniconda3/envs/cwldev/bin/cwltool 1.0.20190228200117
Resolved 'inp.cwl' to 'file:///home/stain/src/cwltool/92/inp.cwl'
[job inp.cwl] initializing from file:///home/stain/src/cwltool/92/inp.cwl
[job inp.cwl] {
"example_file": {
"class": "File",
"location": "file:///home/stain/src/cwltool/92/whale%231.txt",
"basename": "whale.txt",
"size": 0,
"nameroot": "whale",
"nameext": ".txt"
}
}
[job inp.cwl] path mappings is {
"file:///home/stain/src/cwltool/92/whale%231.txt": [
"/home/stain/src/cwltool/92/whale#1.txt",
"/tmp/tmpzs69z374/stgfd7c285d-df95-4036-bdb0-293333725783/whale.txt",
"File",
true
]
}
[job inp.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "cat"
},
{
"position": [
1,
"example_file"
],
"datum": {
"class": "File",
"location": "file:///home/stain/src/cwltool/92/whale%231.txt",
"basename": "whale.txt",
"size": 0,
"nameroot": "whale",
"nameext": ".txt",
"path": "/tmp/tmpzs69z374/stgfd7c285d-df95-4036-bdb0-293333725783/whale.txt",
"dirname": "/tmp/tmpzs69z374/stgfd7c285d-df95-4036-bdb0-293333725783"
}
}
]
[job inp.cwl] initial work dir {}
[job inp.cwl] /tmp/wvgw8rnp$ cat \
/tmp/tmpzs69z374/stgfd7c285d-df95-4036-bdb0-293333725783/whale.txt > /tmp/wvgw8rnp/f500c65a801979809d181d4325cccf1aafa3c49b
Could not collect memory usage, job ended before monitoring began.
[job inp.cwl] completed success
[job inp.cwl] {
"example_out": {
"location": "file:///tmp/wvgw8rnp/f500c65a801979809d181d4325cccf1aafa3c49b",
"basename": "f500c65a801979809d181d4325cccf1aafa3c49b",
"nameroot": "f500c65a801979809d181d4325cccf1aafa3c49b",
"nameext": "",
"class": "File",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"size": 0,
"http://commonwl.org/cwltool#generation": 0
}
}
[job inp.cwl] Removing input staging directory /tmp/tmpzs69z374
[job inp.cwl] Removing temporary directory /tmp/tmpsch2o3xz
Moving /tmp/wvgw8rnp/f500c65a801979809d181d4325cccf1aafa3c49b to /home/stain/src/cwltool/92/f500c65a801979809d181d4325cccf1aafa3c49b
{
"example_out": {
"location": "file:///home/stain/src/cwltool/92/f500c65a801979809d181d4325cccf1aafa3c49b",
"basename": "f500c65a801979809d181d4325cccf1aafa3c49b",
"class": "File",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"size": 0,
"path": "/home/stain/src/cwltool/92/f500c65a801979809d181d4325cccf1aafa3c49b"
}
}
Final process status is success
```
### Missing basename also fails
Without the `basename` we get a similar error as in the command line argument case:
```pytb
(cwldev) stain@biggie:~/src/cwltool/92$ cwltool --debug inp.cwl inp-job-workaround.yml
/home/stain/miniconda3/envs/cwldev/bin/cwltool 1.0.20190228200117
Resolved 'inp.cwl' to 'file:///home/stain/src/cwltool/92/inp.cwl'
[job inp.cwl] initializing from file:///home/stain/src/cwltool/92/inp.cwl
[job inp.cwl] {
"example_file": {
"class": "File",
"location": "file:///home/stain/src/cwltool/92/whale%231.txt",
"size": 0,
"basename": "whale#1.txt",
"nameroot": "whale#1",
"nameext": ".txt"
}
}
Workflow error:
Invalid filename: 'whale#1.txt' contains illegal characters
Traceback (most recent call last):
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/main.py", line 813, in main
logger=_logger)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 39, in __call__
return self.execute(*args, **kwargs)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 101, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/executors.py", line 162, in run_jobs
for job in jobiter:
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 422, in job
visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 217, in visit_class
visit_class(d, cls, op)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/utils.py", line 212, in visit_class
op(rec)
File "/home/stain/miniconda3/envs/cwldev/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 224, in check_adjust
file_o["basename"]))
cwltool.errors.WorkflowException: Invalid filename: 'whale#1.txt' contains illegal characters
```
## Your Environment
* cwltool version: /home/stain/miniconda3/envs/cwldev/bin/cwltool 1.0.20190228200117
Contributor guide
Assessment
This issue has not been assessed yet.