common-workflow-language / common-workflow-language/cwltool
cwltool use of singularity fails with bioconda containers with default output directory (/var/spool/cwl)
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
When cwltool uses a container, it maps a directory on the host filesystem to a working directory path within the container. By default this path is `/var/spool/cwl`. This is passed to the singularity command line where it is used in a bind-mount and for the `--pwd` parameter. For the CWL code below it yields:
```
singularity \
--quiet \
exec \
--contain \
--pid \
--ipc \
--bind \
/tmp/tmpmMeF3o:/var/spool/cwl:rw \
--bind \
/tmp/tmp6fUHbo:/tmp:rw \
--pwd \
/var/spool/cwl \
/some/path/quay.io-biocontainers-blast-2.7.1--boost1.64_1.img \
blastp \
. > /tmp/tmpmMeF3o/listing.txt
```
This is all correct from the `cwltool` side. It leads to a failure with bioconda containers, as described below.
## Actual Behavior
Bioconda containers are built on a `busybox` base container where `/var/spool` is a symbolic link to `/var/tmp` which itself is a symbolic link to `/tmp`.
Singularity does a bind-mount between the host system's `/tmp` and `/tmp` within the container. Because of the symbolic links described above, `/var/spool/cwl` is effectively `/tmp` so the second bind-mount fails (silently). The result is that inside the container `/var/spool/cwl` does not exist, resulting in the error shown in the traceback. I have reported the Singularity issue: https://github.com/singularityware/singularity/issues/1833
A workaround is to use a `dockerRequirement` under `requirements` and specify a `dockerOutputDirectory` that does not clash with an existing bind mount. E.g.
```
requirements:
- class: DockerRequirement
dockerPull: quay.io/biocontainers/blast:2.7.1--boost1.64_1
dockerOutputDirectory: /var/cwl
```
## Workflow Code
```
cwlVersion: v1.0
class: CommandLineTool
hints:
- class: DockerRequirement
dockerPull: quay.io/biocontainers/blast:2.7.1--boost1.64_1
inputs:
filename:
type: string
default: "."
inputBinding:
position: 1
outputs:
output_file:
type: File
outputBinding:
glob: listing.txt
stdout: listing.txt
baseCommand: ls
```
## Full Traceback
```
$ cwltool --singularity example_failure.cwl
/tools/software/toil_venv/bin/cwltool 1.0.20180809224403
Resolved 'example_failure.cwl' to 'file:///usr/people/pvh/mapula/scripts/example_failure.cwl'
Using local copy of Singularity image found in /usr/people/pvh/mapula/scripts
[job example_failure.cwl] /tmp/tmpmMeF3o$ singularity \
--quiet \
exec \
--contain \
--pid \
--ipc \
--bind \
/tmp/tmpmMeF3o:/var/spool/cwl:rw \
--bind \
/tmp/tmp6fUHbo:/tmp:rw \
--pwd \
/var/spool/cwl \
/some/path/quay.io-biocontainers-blast-2.7.1--boost1.64_1.img \
blastp \
. > /tmp/tmpmMeF3o/listing.txt
ERROR : Could not change directory to: /var/spool/cwl
ABORT : Retval = 255
[job example_failure.cwl] completed permanentFail
{
"output_file": {
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"basename": "listing.txt",
"location": "file:///some/path//listing.txt",
"path": "/some/path/listing.txt",
"class": "File",
"size": 0
}
}
Final process status is permanentFail
```
## Your Environment
* cwltool version:
1.0.20180809224403
Contributor guide
Assessment
This issue has not been assessed yet.