common-workflow-language / common-workflow-language/cwltool
Singularity: wrong image path upon initial pull
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
If singularity is used to pull a docker image (`--singularity` flag enabled) for the first time and the environment variables `SINGULARITY_LOCALCACHEDIR` or `SINGULARITY_PULLFOLDER` are set to a non default location, an error occurs.
Here is the cwltool output:
```
Building Singularity image...
Singularity container built: /scratch/.singularity/pull/kerstenbreuer-trim_galore-latest.img
Cleaning up...
Done. Container is at: /scratch/.singularity/pull/kerstenbreuer-trim_galore-latest.img
[job adaptor_trimming_and_qc_post_trim] /scratch/breuerk/cwl_working_dir/tmp_outL_gqQU$ singularity \
--quiet \
exec \
--contain \
--pid \
--ipc \
--bind \
/scratch/breuerk/cwl_working_dir/tmp_outL_gqQU:/var/spool/cwl:rw \
--bind \
/scratch/breuerk/cwl_working_dir/tmpVqNnI7:/tmp:rw \
--bind \
/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/sample1_1.fastq:/var/lib/cwl/stg17d7fc48-95c9-495e-9e96-b780271f51ba/sample1_1.fastq:ro \
--bind \
/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/sample1_2.fastq:/var/lib/cwl/stg2fd55604-fd5d-4316-bf39-6af9568a1968/sample1_2.fastq:ro \
--pwd \
/var/spool/cwl \
/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/kerstenbreuer-trim_galore-latest.img \
trim_galore
```
What happens is that singularity pull an image to the non default directory `/scratch/.singularity/pull/`. Then cwltool is trying to start the image but specifies the wrong path `/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/kerstenbreuer-trim_galore-latest.img`, which is in the current working directory.
Therefore the workflow finally fails with the following error:
```
ESC[91mERROR : Image path /ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/kerstenbreuer-trim_galore-latest.img doesn't exist: No such file or directory
```
However, this error only occurs if the image is pulled for the first time. Upon rerunning the workflow the correct location of the image is recognized.
Singularity is a much better containerization solution than docker for large clusters that are shared by multiple people. We depend on it and would really like to see this fixed soon.
Thank you
Contributor guide
Research direction
Start by tracing cwltool's Singularity image pull and execution path with --singularity enabled, focusing on SINGULARITY_LOCALCACHEDIR and SINGULARITY_PULLFOLDER. Reproduce the first pull using a non-default location and verify that the path passed to singularity exec matches the pulled image; the workflow should also continue to work on a rerun.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100