Document how to successfully use cwltool in docker container

## Environment

```bash
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

$ docker --version
Docker version 19.03.2, build 6a30dfc

$ cwltool --version
/home/ubuntu/.local/bin/cwltool 1.0.20190831161204
```

## My workflow and parameters

```bash
$ ls
workflow_parameters.yml workflow.yml

$ cat workflow.yml
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
fastq:
type: File
nthreads:
type: int?
default: 2
steps:
qc_fastq:
run: https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/tool/fastqc/fastqc.cwl
in:
nthreads: nthreads
fastq: fastq
out:
- qc_result
- stdout
- stderr
outputs:
qc_result:
type: File
outputSource: qc_fastq/qc_result

$ cat workflow_parameters.yml
nthreads: 2 # default value of type "int". (optional)
fastq: # type "File"
class: File
location: https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/test/data/small.ERR034597_1.fastq
```

## Run at my host

```bash
$ cwltool --outdir out1 workflow.cwl workflow_parameters.yml
INFO /home/ubuntu/.local/bin/cwltool 1.0.20190831161204
INFO Resolved 'workflow.cwl' to 'file:///home/ubuntu/cwl-issue/workflow.cwl'
INFO [workflow ] start
INFO [workflow ] starting step qc_fastq
INFO [step qc_fastq] start
INFO [job qc_fastq] /tmp/sp9j02nu$ docker \
run \
-i \
--volume=/tmp/sp9j02nu:/yGkDaS:rw \
--volume=/tmp/e7ztw44l:/tmp:rw \
--volume=/tmp/tmp3hms9jb9:/var/lib/cwl/stg92f3e6b5-846d-4654-8f32-466e8c5baa7c/small.ERR034597_1.fastq:ro \
--workdir=/yGkDaS \
--read-only=true \
--log-driver=none \
--user=1000:1000 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/yGkDaS \
--cidfile=/tmp/iztk7pyn/20190905164414-258222.cid \
quay.io/biocontainers/fastqc:0.11.7--pl5.22.0_2 \
fastqc \
-o \
. \
--threads \
2 \
/var/lib/cwl/stg92f3e6b5-846d-4654-8f32-466e8c5baa7c/small.ERR034597_1.fastq > /tmp/sp9j02nu/fastqc-stdout.log 2> /tmp/sp9j02nu/fastqc-stderr.log
INFO [job qc_fastq] Max memory used: 181MiB
INFO [job qc_fastq] completed success
INFO [step qc_fastq] completed success
INFO [workflow ] completed success
{
"qc_result": {
"class": "File",
"basename": "small.ERR034597_1_fastqc.html",
"location": "file:///home/ubuntu/cwl-issue/out1/small.ERR034597_1_fastqc.html",
"size": 224797,
"checksum": "sha1$74a78b50ed0038646d4d9ce053d6950d09a80baa",
"path": "/home/ubuntu/cwl-issue/out1/small.ERR034597_1_fastqc.html"
}
}
INFO Final process status is success

$ ls out1
small.ERR034597_1_fastqc.html
```

- Running, but `--volume=/tmp/tmp3hms9jb9:/var/lib/cwl/stg92f3e6b5-846d-4654-8f32-466e8c5baa7c/small.ERR034597_1.fastq:ro \` looks strange.

```bash
$ head /tmp/tmp3hms9jb9@ERR034597.1 FCB06A1ABXX:6:1101:1083:1974#ACTTGAAT/1
NAGACAACACCGCTGCAGTGCACACGCAGCACGTGGGCTTCAACCGGCAGGAGCAGGACGTGTTCTTCCTGCCCATCCTGGTGGTAGACA
+
#163344445DDDDDDDBDDDDD=D=@DD=9???=@@@@@@@@@@9=@@@<4<:955555:>6:69<99999???###############
@ERR034597.2 FCB06A1ABXX:6:1101:1068:1995#ACTTGAAT/1
GTGCTTTTTTGGTACCACGAGACCCAGCTCTCTGCCAGCATCAACCACTCACACCCCAGGGGGATGAGCTCTCCAAGCCCAGGATTCTGG
+
HHHHHHHHHHHHHHHGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHGHHHHHHHHHHHHFEEFEFEHHHHHHEHHHHFFHFHHFHEEDDD
@ERR034597.3 FCB06A1ABXX:6:1101:1190:1997#ACTTGAAT/1
TAAATCCAAACTATAGCTGTCGTTTTTATCATTTCCCAATACAGAAGCTTTTCTTTTGCTTGTTCGTAATGCTACACTTTTTGTTCAGCT
```

## Run in docker container

```bash
$ docker run -it --rm \
-v $PWD:$PWD --workdir $PWD \
-v /usr/bin/docker:/usr/bin/docker \
-v /var/lib/docker:/var/lib/docker \
-v /var/run/docker.sock:/var/run/docker.sock \
python:slim bash

root@83b92eadaab9:/home/ubuntu/cwl-issue# apt update
root@83b92eadaab9:/home/ubuntu/cwl-issue# apt install -y build-essential
root@83b92eadaab9:/home/ubuntu/cwl-issue# pip3 install cwltool==1.0.20190831161204
root@83b92eadaab9:/home/ubuntu/cwl-issue# cwltool --version
/usr/local/bin/cwltool 1.0.20190831161204

root@83b92eadaab9:/home/ubuntu/cwl-issue# cwltool --outdir out2 --debug workflow.cwl workflow_parameters.yml
INFO /usr/local/bin/cwltool 1.0.20190831161204
INFO Resolved 'workflow.cwl' to 'file:///home/ubuntu/cwl-issue/workflow.cwl'
INFO [workflow ] start
INFO [workflow ] starting step qc_fastq
INFO [step qc_fastq] start
INFO [job qc_fastq] /tmp/8oww15py$ docker \
run \
-i \
--volume=/tmp/8oww15py:/UdfqtH:rw \
--volume=/tmp/p78sg0zs:/tmp:rw \
--volume=/tmp/tmp_897xhr9:/var/lib/cwl/stgcb797c08-7af2-48cd-86cc-39e88dd7a5cd/small.ERR034597_1.fastq:ro \
--workdir=/UdfqtH \
--read-only=true \
--log-driver=none \
--user=0:0 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/UdfqtH \
--cidfile=/tmp/h6e_cnkk/20190905075202-759019.cid \
quay.io/biocontainers/fastqc:0.11.7--pl5.22.0_2 \
fastqc \
-o \
. \
--threads \
2 \
/var/lib/cwl/stgcb797c08-7af2-48cd-86cc-39e88dd7a5cd/small.ERR034597_1.fastq > /tmp/8oww15py/fastqc-stdout.log 2> /tmp/8oww15py/fastqc-stderr.log
INFO [job qc_fastq] Max memory used: 0MiB
ERROR [job qc_fastq] Job error:
("Error collecting output for parameter 'qc_result':\nhttps://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/tool/fastqc/fastqc.cwl:27:7: Did not find output file with glob pattern: '['*_fastqc.html']'", {})
WARNING [job qc_fastq] completed permanentFail
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/qc_result
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/stdout
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/stderr
WARNING [step qc_fastq] completed permanentFail
INFO [workflow ] completed permanentFail
{
"qc_result": null
}
WARNING Final process status is permanentFail
```

- Failed
- Use debug mode

```bash
root@83b92eadaab9:/home/ubuntu/cwl-issue# cwltool --outdir out2 --debug workflow.cwl workflow_parameters.yml
INFO /usr/local/bin/cwltool 1.0.20190831161204
INFO Resolved 'workflow.cwl' to 'file:///home/ubuntu/cwl-issue/workflow.cwl'
DEBUG [workflow ] initialized from file:///home/ubuntu/cwl-issue/workflow.cwl
INFO [workflow ] start
DEBUG [workflow ] {
"nthreads": 2,
"fastq": {
"class": "File",
"location": "https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/test/data/small.ERR034597_1.fastq",
"basename": "small.ERR034597_1.fastq",
"nameroot": "small.ERR034597_1",
"nameext": ".fastq"
}
}
INFO [workflow ] starting step qc_fastq
DEBUG [job step qc_fastq] job input {
"file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/fastq": {
"class": "File",
"location": "https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/test/data/small.ERR034597_1.fastq",
"basename": "small.ERR034597_1.fastq",
"nameroot": "small.ERR034597_1",
"nameext": ".fastq"
},
"file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/nthreads": 2
}
DEBUG [job step qc_fastq] evaluated job input to {
"file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/fastq": {
"class": "File",
"location": "https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/test/data/small.ERR034597_1.fastq",
"basename": "small.ERR034597_1.fastq",
"nameroot": "small.ERR034597_1",
"nameext": ".fastq"
},
"file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/nthreads": 2
}
INFO [step qc_fastq] start
DEBUG [job qc_fastq] initializing from https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/tool/fastqc/fastqc.cwl as part of step qc_fastq
DEBUG [job qc_fastq] {
"fastq": {
"class": "File",
"location": "https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/test/data/small.ERR034597_1.fastq",
"basename": "small.ERR034597_1.fastq",
"nameroot": "small.ERR034597_1",
"nameext": ".fastq"
},
"nthreads": 2
}
DEBUG [job qc_fastq] path mappings is {
"https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/test/data/small.ERR034597_1.fastq": [
"/tmp/tmpn8_8f3p1",
"/var/lib/cwl/stg34337193-c77a-46a3-83a5-2bcec727c374/small.ERR034597_1.fastq",
"File",
true
]
}
DEBUG [job qc_fastq] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "fastqc"
},
{
"position": [
0,
0
],
"prefix": "-o",
"valueFrom": "."
},
{
"position": [
1,
"nthreads"
],
"prefix": "--threads",
"datum": 2
},
{
"position": [
2,
"fastq"
],
"datum": {
"class": "File",
"location": "https://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/test/data/small.ERR034597_1.fastq",
"basename": "small.ERR034597_1.fastq",
"nameroot": "small.ERR034597_1",
"nameext": ".fastq",
"path": "/var/lib/cwl/stg34337193-c77a-46a3-83a5-2bcec727c374/small.ERR034597_1.fastq",
"dirname": "/var/lib/cwl/stg34337193-c77a-46a3-83a5-2bcec727c374"
}
}
]
DEBUG [job qc_fastq] initial work dir {}
INFO [job qc_fastq] /tmp/xn_d08oe$ docker \
run \
-i \
--volume=/tmp/xn_d08oe:/cXBLdZ:rw \
--volume=/tmp/o2hv7x4j:/tmp:rw \
--volume=/tmp/tmpn8_8f3p1:/var/lib/cwl/stg34337193-c77a-46a3-83a5-2bcec727c374/small.ERR034597_1.fastq:ro \
--workdir=/cXBLdZ \
--read-only=true \
--log-driver=none \
--user=0:0 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/cXBLdZ \
--cidfile=/tmp/lfpf38_s/20190905075307-913057.cid \
quay.io/biocontainers/fastqc:0.11.7--pl5.22.0_2 \
fastqc \
-o \
. \
--threads \
2 \
/var/lib/cwl/stg34337193-c77a-46a3-83a5-2bcec727c374/small.ERR034597_1.fastq > /tmp/xn_d08oe/fastqc-stdout.log 2> /tmp/xn_d08oe/fastqc-stderr.log
INFO [job qc_fastq] Max memory used: 0MiB
ERROR [job qc_fastq] Job error:
("Error collecting output for parameter 'qc_result':\nhttps://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/tool/fastqc/fastqc.cwl:27:7: Did not find output file with glob pattern: '['*_fastqc.html']'", {})
WARNING [job qc_fastq] completed permanentFail
DEBUG [job qc_fastq] {}
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/qc_result
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/stdout
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/stderr
DEBUG [step qc_fastq] produced output {}
WARNING [step qc_fastq] completed permanentFail
INFO [workflow ] completed permanentFail
DEBUG [workflow ] {
"qc_result": null
}
DEBUG [job qc_fastq] Removing input staging directory /tmp/avk5jogy
DEBUG [job qc_fastq] Removing temporary directory /tmp/o2hv7x4j
{
"qc_result": null
}
WARNING Final process status is permanentFail
```

## Use cwltool official container

```bash
$ git clone https://github.com/common-workflow-language/cwltool.git
$ cd cwltool
$ docker build -f cwltool.Dockerfile -t cwltool:latest .
$ cd ..
$ docker run -it --rm \
-v $PWD:$PWD --workdir $PWD \
-v /var/run/docker.sock:/var/run/docker.sock \
--entrypoint "" \
cwltool:latest sh

/home/ubuntu/cwl-issue # cwltool --version
/usr/local/bin/cwltool 1.0.20190905070411
/home/ubuntu/cwl-issue # cwltool --outdir out2 workflow.cwl workflow_paramete
rs.yml
INFO /usr/local/bin/cwltool 1.0.20190905070411
INFO Resolved 'workflow.cwl' to 'file:///home/ubuntu/cwl-issue/workflow.cwl'
INFO [workflow ] start
INFO [workflow ] starting step qc_fastq
INFO [step qc_fastq] start
INFO [job qc_fastq] /tmp/t3v4_hft$ docker \
run \
-i \
--volume=/tmp/t3v4_hft:/KaVxCR:rw \
--volume=/tmp/zgnw2632:/tmp:rw \
--volume=/tmp/tmpss22aac3:/var/lib/cwl/stg9eeea4e8-7b79-4cfb-8fea-5358c1ec1f5e/small.ERR034597_1.fastq:ro \
--workdir=/KaVxCR \
--read-only=true \
--log-driver=none \
--user=0:0 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/KaVxCR \
--cidfile=/tmp/vhsktbpm/20190905080855-408247.cid \
quay.io/biocontainers/fastqc:0.11.7--pl5.22.0_2 \
fastqc \
-o \
. \
--threads \
2 \
/var/lib/cwl/stg9eeea4e8-7b79-4cfb-8fea-5358c1ec1f5e/small.ERR034597_1.fastq > /tmp/t3v4_hft/fastqc-stdout.log 2> /tmp/t3v4_hft/fastqc-stderr.log
INFO [job qc_fastq] Max memory used: 0MiB
ERROR [job qc_fastq] Job error:
("Error collecting output for parameter 'qc_result':\nhttps://raw.githubusercontent.com/suecharo/SAPPORO_test_workflow/master/tool/fastqc/fastqc.cwl:27:7: Did not find output file with glob pattern: '['*_fastqc.html']'", {})
WARNING [job qc_fastq] completed permanentFail
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/qc_result
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/stdout
ERROR [step qc_fastq] Output is missing expected field file:///home/ubuntu/cwl-issue/workflow.cwl#qc_fastq/stderr
WARNING [step qc_fastq] completed permanentFail
INFO [workflow ] completed permanentFail
{
"qc_result": null
}
WARNING Final process status is permanentFail
```

- same probrem