conda-forge / conda-forge/conda-forge.github.io

Testing packages in minimal containers and creating containers on the fly

Open
#561 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
170
Forks
320
Avg merge
2d 10h
Merged PRs (30d)
5

Description

Hi all.

@jakirkham asks me to write about our work on Containers in https://github.com/conda-forge/staged-recipes/pull/4838#discussion_r180283102

Here we go:

In Bioconda we are using a busybox container to test all our recipes. This is very useful for catching errors like missing zlib or missing libgcc for example and it ensures it runs everywhere. As a nice side-effect we are building optimized and minimal containers. Many containers are just a few MB in size.

A preprint about this project can be found here: https://www.biorxiv.org/content/early/2017/10/11/200683 (biology motivated, but the technical concept is universal)

We have created to nearly 3000 packages containers, with multiple versions.
Please feel free to ask any question and provide feedback.

## Single package containers:

These are easy. We are using a build-container with miniconda installed to install (the previously locally build package from the local channel) into the build-container. We than move everything that is needed to the runtime container - in this case busybox - and we give the container a predictable name. So that everyone can `guess` the name without knowing that there is a container.

## Multiple package containers:

More complicated. We need to create a predictable name-space for the containers. What we do is `mulled-v2-`:``. Otherwise, same procedure as the single-package-containers. We use a build and a runtime container. We do *not* use a Dockerfile in both cases.

Try this one out to get a feeling how it works: http://biocontainers.pro/multi-package-containers/

Create multiple package containers with simple PRs like this one: https://github.com/BioContainers/multi-package-containers/pull/427

## Tooling

What happens underneath is documented here a little bit:
https://docs.galaxyproject.org/en/master/admin/mulled_containers.html#automatic-build-of-linux-containers

## Containers

Docker/rkt containers are here: https://quay.io/organization/biocontainers
Singularity container are here: https://depot.galaxyproject.org/singularity/

## Misc

More container engines can be easily added, indeed we have plans for swifter etc.
Creating the Docker/rkt containers happens during the CI step of Bioconda for Linux and are also pushed from within the CI.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.