nextflow-io / nextflow-io/nextflow

singularity.cacheDir does not work properly with containers that have ':' in the filename

Open
#5,324 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

software/singularity stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

with my nextflow.config like this;


singularity {
    enabled = true
    autoMounts = true
    cacheDir = "/fsx/home/tools/containers"
}

process {
    executor = 'slurm'
    container = 'nextflow_bash:latest.sif'

    withName: DebugScript {
        container = 'pytorch_pytorch:latest.sif'
    }
}


The required containers exist as such;

$ ll /fsx/home/tools/containers/nextflow_bash:latest.sif
-rw-r--r-- 1 username mygroup 29M Sep 23 16:44 /fsx/home/tools/containers/nextflow_bash:latest.sif

$ ll /fsx/home/tools/containers/pytorch_pytorch:latest.sif
-rw-r--r-- 1 username mygroup 3.5G Sep 23 17:39 /fsx/home/tools/containers/pytorch_pytorch:latest.sif

However, when you run Nextflow, it breaks with an error such as

Pulling Singularity image docker://nextflow_bash:latest.sif [cache /fsx/home/tools/containers/nextflow_bash-latest.sif]
Pulling Singularity image docker://pytorch_pytorch:latest.sif [cache /fsx/home/tools/containers/pytorch_pytorch-latest.sif]
ERROR ~ Error executing process > 'DebugScript'

Caused by:
  Failed to pull singularity image
    command: singularity pull  --name pytorch_pytorch-latest.sif.pulling.1727127729010 docker://pytorch_pytorch:latest.sif > /dev/null
    status : 255

As you can see from the log and error messages, the cacheDir directive is not correctly detecting the files nextflow_bash:latest.sif and pytorch_pytorch:latest.sif, and is instead looking for the files nextflow_bash-latest.sif and pytorch_pytorch-latest.sif which do not exist.

Is this a bug? This behavior is not described in the docs ( https://www.nextflow.io/docs/latest/config.html#scope-singularity ), and I have not encountered this issue in the past with filenames that lack ':'. Considering that our files are on a soon-to-be S3 backed storage system, having to constantly rename the .sif files to match this unexpected different filename is a little bothersome.  

Environment
  • Nextflow version: nextflow version 24.04.4.5917

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Singularity cacheDir handling described in the configuration scope and the pull commands shown in the report. Compare cache lookup and pull naming for container filenames containing ':'; done means existing colon-containing .sif files are detected without being renamed, with the relevant tests passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.