nf-core / nf-core/tools

Use singularity containers for apptainer profile too

Open
#2,357 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
322
Forks
255
Avg merge
2d 3h
Merged PRs (30d)
5

Description

Description of feature

Singularity has split out into an open source version called Apptainer two years ago.

Currently the logic for using singularity containers for modules is hardcoded to require the use of the singularity container profile, e.g.

 container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://depot.galaxyproject.org/singularity/ubuntu:20.04' :
        'nf-core/ubuntu:20.04' }"

so the apptainer profile will build from the docker containers instead, which is less robust and more likely to fail. It would therefore be good to enable the singularity images for apptainer too.

This was discussed in on Slack, with this summary from Phil @ewels :

Yup, so in summary:
Short term fix to just get things working is apptainer.registry, now merged and will go out in release imminently
TODO: Medium term fix: We should write up some docs explaining about the singularity command alias and how we can trick the system to work as you described.
TODO: Long term fix: Update logic in all modules, plus all container parsing logic, plus all infra, to handle apptainer profile name 🙈
May be worth discussing using a param instead of relying on workflow.containerEngine for better future proofing / more concise code?
ℹ️ Note that Nextflow will hopefully give us a list of container names soon, meaning a significant rewrite for the nf-core download infra, may make sense to wait for that and do this all in one go

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the module container logic, container parsing logic, and infrastructure handling for the singularity and apptainer profiles. Review the linked Slack discussion and the existing apptainer.registry change before defining the scope. Done means the relevant modules and profile infrastructure consistently support singularity images with Apptainer.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.