conda-forge / conda-forge/conda-forge.github.io

Testing packages in minimal containers and creating containers on the fly

Aperta
#561 0 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
JavaScript
Stelle
170
Fork
320
Merge medio
2g 10h
PR unite (30g)
5

Descrizione

Hi all.

@jakirkham asks me to write about our work on Containers in https://github.com/conda-forge/staged-recipes/pull/4838#discussion_r180283102

Here we go:

In Bioconda we are using a busybox container to test all our recipes. This is very useful for catching errors like missing zlib or missing libgcc for example and it ensures it runs everywhere. As a nice side-effect we are building optimized and minimal containers. Many containers are just a few MB in size.

A preprint about this project can be found here: https://www.biorxiv.org/content/early/2017/10/11/200683 (biology motivated, but the technical concept is universal)

We have created to nearly 3000 packages containers, with multiple versions.
Please feel free to ask any question and provide feedback.

## Single package containers:

These are easy. We are using a build-container with miniconda installed to install (the previously locally build package from the local channel) into the build-container. We than move everything that is needed to the runtime container - in this case busybox - and we give the container a predictable name. So that everyone can `guess` the name without knowing that there is a container.

## Multiple package containers:

More complicated. We need to create a predictable name-space for the containers. What we do is `mulled-v2-`:``. Otherwise, same procedure as the single-package-containers. We use a build and a runtime container. We do *not* use a Dockerfile in both cases.

Try this one out to get a feeling how it works: http://biocontainers.pro/multi-package-containers/

Create multiple package containers with simple PRs like this one: https://github.com/BioContainers/multi-package-containers/pull/427

## Tooling

What happens underneath is documented here a little bit:
https://docs.galaxyproject.org/en/master/admin/mulled_containers.html#automatic-build-of-linux-containers

## Containers

Docker/rkt containers are here: https://quay.io/organization/biocontainers
Singularity container are here: https://depot.galaxyproject.org/singularity/

## Misc

More container engines can be easily added, indeed we have plans for swifter etc.
Creating the Docker/rkt containers happens during the CI step of Bioconda for Linux and are also pushed from within the CI.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.