common-workflow-language / common-workflow-language/cwltool
File literal content is used directly as temporary filename
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
A file literal (`class: File` with `content`) in the job file should be written to file, which filename should be autogenerated or based on `basename`
## Actual Behavior
A file is created using the `contents` value as a file name, which is then passed in as a docker `--volume` and mounted with a more reasonable `basename` inside.
The filename on the host can therefore be horrendously long and include newlines, which is very confusing. There is also the potential that some characters in `contents` are not valid in the filename, e.g. `/` and `:`.
## Workflow Code
### sed-job.yml
```cwl
(base) stain@biggie:~/src/ro-index-paper/code/data-gathering/test/sed$ cat sed-job.yml
#!/usr/bin/env cwltool
cwl:tool: sed.cwl
command: s/fred/soup/g
original:
class: File
# basename: "hello.txt"
contents: |
This is a hello world file where fred is replaced with soup, and even fred
is changed to soup multiple times, just like Fred said fred should be.
```
### sed.cwl
```cwl
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
# Copyright 2019 The University of Manchester
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
baseCommand: "sed"
hints:
SoftwareRequirement:
packages:
sed:
specs:
- https://anaconda.org/conda-forge/sed
- https://packages.debian.org/jessie/sed
- https://packages.debian.org/stretch/sed
- https://packages.debian.org/buster/sed
- https://packages.debian.org/bullseye/sed
- https://packages.debian.org/sid/sed
version: [ "4.2.2", "4.4", "4.7"] # .. and many more
DockerRequirement:
dockerPull: debian:9
arguments:
- "--unbuffered"
- "--regexp-extended"
inputs:
original:
type: File
streamable: true
doc: >
The original file or byte stream which content is to be modified
inputBinding:
position: 2
command:
type: string
doc: |
The regular expression to be used for search-replace,
in the form of a sed "s/from/to/" command, e.g.
"s/^/_/" will insert an underscore at beginning of each line,
while "s/http/https/g" will replace every "http" with "https".
This CWL wrapper enables the extended (ERE) regular expression
and operates in streamable mode.
See also
and
inputBinding:
position: 1
outputs:
modified:
type: stdout
streamable: true
doc: The modified text file
#stdin: original.txt
stdout: modified.txt
label: "sed search-replace"
doc: >
Search-replace a stream using regular expressions and other SED commands.
s:author:
- class: s:Person
s:name: Jay Fenlason
- class: s:Person
s:name: Tom Lord
- class: s:Person
s:name: Ken Pizzini
- class: s:Person
s:name: Paolo Bonzini
s:copyrightHolder:
- class: s:Organization
s:name: Free Software Foundation
s:url: https://www.fsf.org/
s:url: https://www.gnu.org/software/sed/
s:mainEntityOfPage: https://www.gnu.org/software/sed/manual/
s:codeRepository: https://github.com/LibreCat/perl-oai-lib/
s:license: https://spdx.org/licenses/GPL-3.0-or-later
s:sdLicense: https://spdx.org/licenses/Apache-2.0
s:sdPublisher: https://orcid.org/0000-0001-9842-9718
s:description: >
sed (stream editor) is a non-interactive command-line text editor.
sed is commonly used to filter text, i.e., it takes text input, performs
some operation (or set of operations) on it, and outputs the modified text.
sed is typically used for extracting part of a file using pattern matching or
substituting multiple occurrences of a string within a file.
s:potentialAction:
- class: s:ActivateAction
s:label: "example run"
s:instrument: "../test/sed-job.yml"
$schemas:
- https://schema.org/version/3.9/schema.rdf
$namespaces:
iana: https://www.iana.org/assignments/media-types/
s: https://schema.org/
```
## Full Traceback
```
(base) stain@biggie:~/src/ro-index-paper/code/data-gathering/test/sed$ ./sed-job.yml
INFO /home/stain/miniconda3/bin/cwltool 1.0.20190815141648
INFO Resolved './sed-job.yml' to 'file:///home/stain/src/ro-index-paper/code/data-gathering/test/sed/sed-job.yml'
INFO [job sed.cwl] /tmp/syd88933$ docker \
run \
-i \
--volume=/tmp/syd88933:/hvvYnS:rw \
--volume=/tmp/x2k9yb34:/tmp:rw \
'--volume=/tmp/3a0gxqpb/This is a hello world file where fred is replaced with soup, and even fred
is changed to soup multiple times, just like Fred said fred should be.
:/var/lib/cwl/stg35cc32fa-4801-435f-8c84-192259c2ee4f/cdbc1583-95bd-4f4a-82ae-5e77b0ceb0b9:ro' \
--workdir=/hvvYnS \
--read-only=true \
--log-driver=none \
--user=1000:1000 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/hvvYnS \
--cidfile=/tmp/9_l7lajo/20190821141917-474313.cid \
debian:9 \
sed \
--unbuffered \
--regexp-extended \
s/fred/soup/g \
/var/lib/cwl/stg35cc32fa-4801-435f-8c84-192259c2ee4f/cdbc1583-95bd-4f4a-82ae-5e77b0ceb0b9 > /tmp/syd88933/modified.txt
INFO [job sed.cwl] Max memory used: 0MiB
INFO [job sed.cwl] completed success
{
"modified": {
"location": "file:///home/stain/src/ro-index-paper/code/data-gathering/test/sed/modified.txt",
"basename": "modified.txt",
"class": "File",
"checksum": "sha1$d0dbd8d4fb24b8de348c0193f75f95b6d9f9d6a6",
"size": 146,
"path": "/home/stain/src/ro-index-paper/code/data-gathering/test/sed/modified.txt"
}
}
INFO Final process status is success
```
It seems to work because the content of the file somewhat matches its own filename
```
(base) stain@biggie:/tmp/3a0gxqpb$ ls -al
total 40
drwx------ 2 stain stain 4096 Aug 21 14:19 .
drwxrwxrwt 267 root root 28672 Aug 21 14:25 ..
-r--r--r-- 1 stain stain 146 Aug 21 14:19 'This is a hello world file where fred is replaced with soup, and even fred'$'\n''is changed to soup multiple times, just like Fred said fred should be.'$'\n'
(base) stain@biggie:/tmp/3a0gxqpb$ cat T*
This is a hello world file where fred is replaced with soup, and even fred
is changed to soup multiple times, just like Fred said fred should be.
```
## Your Environment
* cwltool version:
Check using ``cwltool --version``
/home/stain/miniconda3/bin/cwltool 1.0.20190815141648
Contributor guide
Assessment
This issue has not been assessed yet.