common-workflow-language / common-workflow-language/cwltool

Singularity: wrong image path upon initial pull

Open
#889 0 comments 5 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

If singularity is used to pull a docker image (`--singularity` flag enabled) for the first time and the environment variables `SINGULARITY_LOCALCACHEDIR` or `SINGULARITY_PULLFOLDER` are set to a non default location, an error occurs.

Here is the cwltool output:
```
Building Singularity image...
Singularity container built: /scratch/.singularity/pull/kerstenbreuer-trim_galore-latest.img
Cleaning up...
Done. Container is at: /scratch/.singularity/pull/kerstenbreuer-trim_galore-latest.img
[job adaptor_trimming_and_qc_post_trim] /scratch/breuerk/cwl_working_dir/tmp_outL_gqQU$ singularity \
--quiet \
exec \
--contain \
--pid \
--ipc \
--bind \
/scratch/breuerk/cwl_working_dir/tmp_outL_gqQU:/var/spool/cwl:rw \
--bind \
/scratch/breuerk/cwl_working_dir/tmpVqNnI7:/tmp:rw \
--bind \
/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/sample1_1.fastq:/var/lib/cwl/stg17d7fc48-95c9-495e-9e96-b780271f51ba/sample1_1.fastq:ro \
--bind \
/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/sample1_2.fastq:/var/lib/cwl/stg2fd55604-fd5d-4316-bf39-6af9568a1968/sample1_2.fastq:ro \
--pwd \
/var/spool/cwl \
/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/kerstenbreuer-trim_galore-latest.img \
trim_galore
```
What happens is that singularity pull an image to the non default directory `/scratch/.singularity/pull/`. Then cwltool is trying to start the image but specifies the wrong path `/ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/kerstenbreuer-trim_galore-latest.img`, which is in the current working directory.

Therefore the workflow finally fails with the following error:
```
ESC[91mERROR : Image path /ngs_share/pipelines/ATAC_ChIP_pipelines/tiny_test/kerstenbreuer-trim_galore-latest.img doesn't exist: No such file or directory
```

However, this error only occurs if the image is pulled for the first time. Upon rerunning the workflow the correct location of the image is recognized.

Singularity is a much better containerization solution than docker for large clusters that are shared by multiple people. We depend on it and would really like to see this fixed soon.

Thank you

Contributor guide

Open the contributing guide

Research direction

Start by tracing cwltool's Singularity image pull and execution path with --singularity enabled, focusing on SINGULARITY_LOCALCACHEDIR and SINGULARITY_PULLFOLDER. Reproduce the first pull using a non-default location and verify that the path passed to singularity exec matches the pulled image; the workflow should also continue to work on a rerun.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.