conda-forge / conda-forge/conda-forge.github.io
Testing packages in minimal containers and creating containers on the fly
- Dominant language
- JavaScript
- Stars
- 170
- Forks
- 320
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 5
Description
Hi all.
@jakirkham asks me to write about our work on Containers in https://github.com/conda-forge/staged-recipes/pull/4838#discussion_r180283102
Here we go:
In Bioconda we are using a busybox container to test all our recipes. This is very useful for catching errors like missing zlib or missing libgcc for example and it ensures it runs everywhere. As a nice side-effect we are building optimized and minimal containers. Many containers are just a few MB in size.
A preprint about this project can be found here: https://www.biorxiv.org/content/early/2017/10/11/200683 (biology motivated, but the technical concept is universal)
We have created to nearly 3000 packages containers, with multiple versions.
Please feel free to ask any question and provide feedback.
## Single package containers:
These are easy. We are using a build-container with miniconda installed to install (the previously locally build package from the local channel) into the build-container. We than move everything that is needed to the runtime container - in this case busybox - and we give the container a predictable name. So that everyone can `guess` the name without knowing that there is a container.
## Multiple package containers:
More complicated. We need to create a predictable name-space for the containers. What we do is `mulled-v2-`:``. Otherwise, same procedure as the single-package-containers. We use a build and a runtime container. We do *not* use a Dockerfile in both cases.
Try this one out to get a feeling how it works: http://biocontainers.pro/multi-package-containers/
Create multiple package containers with simple PRs like this one: https://github.com/BioContainers/multi-package-containers/pull/427
## Tooling
What happens underneath is documented here a little bit:
https://docs.galaxyproject.org/en/master/admin/mulled_containers.html#automatic-build-of-linux-containers
## Containers
Docker/rkt containers are here: https://quay.io/organization/biocontainers
Singularity container are here: https://depot.galaxyproject.org/singularity/
## Misc
More container engines can be easily added, indeed we have plans for swifter etc.
Creating the Docker/rkt containers happens during the CI step of Bioconda for Linux and are also pushed from within the CI.
Contributor guide
Assessment
This issue has not been assessed yet.