galaxyproject / galaxyproject/galaxy

Containers created with involucro are not found

Open
#6,071 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
169

Description

Hi all,
I have a small issue when trying to run tools using docker dependencies.
It works perfectly when there is only one dependency and it is found and pulled from biocontainers.

But for cases when there are several dependencies....
**it can't find the mulled container, which is right, then starts correctly to create the image using involucro:**

```
galaxy.tools.deps.containers INFO 2018-05-07 13:42:31,460 Checking with container resolver [ExplicitContainerResolver[]] found description [None]
docker images
galaxy.tools.deps.containers INFO 2018-05-07 13:42:31,499 Checking with container resolver [CachedMulledDockerContainerResolver[namespace=biocontainers]] found description [None]
requests.packages.urllib3.connectionpool INFO 2018-05-07 13:42:31,503 Starting new HTTPS connection (1): quay.io
requests.packages.urllib3.connectionpool DEBUG 2018-05-07 13:42:32,125 "GET /api/v1/repository/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 HTTP/1.1" 200 1355
galaxy.tools.deps.containers INFO 2018-05-07 13:42:32,127 Checking with container resolver [MulledDockerContainerResolver[namespace=biocontainers]] found description [None]
./involucro -v=3 -f /galaxy_base/src_prod/lib/galaxy/tools/deps/mulled/invfile.lua -set CHANNELS='bioconda,conda-forge,r' -set TEST='true' -set TARGETS='bwa=0.7.17,samtools=1.8' -set REPO='quay.io/local/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:23592e4ad15ca2acfca18facab87a1ce22c49da1-0' -set BINDS='build/dist:/usr/local/' build-and-test

**The involucro build-and-test seems to work ok:**

[May 7 13:42:32] DEBU Run file [/galaxy_base/src_prod/lib/galaxy/tools/deps/mulled/invfile.lua]
[May 7 13:42:32] INFO Invoke Task [build]
[May 7 13:42:32] STEP Run image [continuumio/miniconda:latest] with command [[rm -rf /data/dist]]
[May 7 13:42:32] DEBU Creating container [step-0733aca0f7]
[May 7 13:42:32] DEBU Created container [e1ab0fca2d43 step-0733aca0f7], starting it
[May 7 13:42:37] DEBU Container [e1ab0fca2d43 step-0733aca0f7] started, waiting for completion
[May 7 13:42:37] DEBU Container [e1ab0fca2d43 step-0733aca0f7] completed with exit code [0] as expected
[May 7 13:42:37] DEBU Container [e1ab0fca2d43 step-0733aca0f7] removed
[May 7 13:42:37] STEP Run image [continuumio/miniconda:latest] with command [[/bin/sh -c conda install -c bioconda -c conda-forge -c r bwa=0.7.17 samtools=1.8 -p /usr/local --copy --yes --quiet]]
[May 7 13:42:37] DEBU Creating container [step-57cca42fca]
[May 7 13:42:38] DEBU Created container [a9b07ae5f8ab step-57cca42fca], starting it
[May 7 13:42:40] DEBU Container [a9b07ae5f8ab step-57cca42fca] started, waiting for completion
[May 7 13:43:23] SOUT Solving environment: ...working... done
[May 7 13:43:23] SERR
[May 7 13:43:23] SERR
[May 7 13:43:23] SERR ==> WARNING: A newer version of conda exists. <==
[May 7 13:43:23] SERR current version: 4.4.10
[May 7 13:43:23] SERR latest version: 4.5.2
[May 7 13:43:23] SERR
[May 7 13:43:23] SERR Please update conda by running
[May 7 13:43:23] SERR
[May 7 13:43:23] SERR $ conda update -n base conda
[May 7 13:43:23] SERR
[May 7 13:43:23] SERR
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT ## Package Plan ##
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT environment location: /usr/local
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT added / updated specs:
[May 7 13:44:48] SOUT - bwa=0.7.17
[May 7 13:44:48] SOUT - samtools=1.8
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT The following packages will be downloaded:
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT package | build
[May 7 13:44:48] SOUT ---------------------------|-----------------
[May 7 13:44:48] SOUT openssl-1.0.2o | 0 3.5 MB conda-forge
[May 7 13:44:48] SOUT zlib-1.2.11 | 0 93 KB conda-forge
[May 7 13:44:48] SOUT ncurses-5.9 | 10 1.1 MB conda-forge
[May 7 13:44:48] SOUT libdeflate-0.8 | 0 38 KB bioconda
[May 7 13:44:48] SOUT libgcc-ng-7.2.0 | hdf63c60_3 6.1 MB
[May 7 13:44:48] SOUT samtools-1.8 | 3 1.7 MB bioconda
[May 7 13:44:48] SOUT perl-5.22.0.1 | 0 15.1 MB conda-forge
[May 7 13:44:48] SOUT curl-7.59.0 | 1 848 KB conda-forge
[May 7 13:44:48] SOUT ca-certificates-2018.4.16 | 0 139 KB conda-forge
[May 7 13:44:48] SOUT krb5-1.14.6 | 0 4.0 MB conda-forge
[May 7 13:44:48] SOUT libgcc-7.2.0 | h69d50b8_2 304 KB
[May 7 13:44:48] SOUT bzip2-1.0.6 | 1 310 KB conda-forge
[May 7 13:44:48] SOUT xz-5.2.3 | 0 854 KB conda-forge
[May 7 13:44:48] SOUT libssh2-1.8.0 | 2 243 KB conda-forge
[May 7 13:44:48] SOUT bwa-0.7.17 | pl5.22.0_2 502 KB bioconda
[May 7 13:44:48] SOUT libstdcxx-ng-7.2.0 | hdf63c60_3 2.5 MB
[May 7 13:44:48] SOUT ------------------------------------------------------------
[May 7 13:44:48] SOUT Total: 37.3 MB
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT The following NEW packages will be INSTALLED:
[May 7 13:44:48] SOUT
[May 7 13:44:48] SOUT bwa: 0.7.17-pl5.22.0_2 bioconda
[May 7 13:44:48] SOUT bzip2: 1.0.6-1 conda-forge
[May 7 13:44:48] SOUT ca-certificates: 2018.4.16-0 conda-forge
[May 7 13:44:48] SOUT curl: 7.59.0-1 conda-forge
[May 7 13:44:48] SOUT krb5: 1.14.6-0 conda-forge
[May 7 13:44:48] SOUT libdeflate: 0.8-0 bioconda
[May 7 13:44:48] SOUT libgcc: 7.2.0-h69d50b8_2
[May 7 13:44:48] SOUT libgcc-ng: 7.2.0-hdf63c60_3
[May 7 13:44:48] SOUT libssh2: 1.8.0-2 conda-forge
[May 7 13:44:48] SOUT libstdcxx-ng: 7.2.0-hdf63c60_3
[May 7 13:44:48] SOUT ncurses: 5.9-10 conda-forge
[May 7 13:44:48] SOUT openssl: 1.0.2o-0 conda-forge
[May 7 13:44:48] SOUT perl: 5.22.0.1-0 conda-forge
[May 7 13:44:48] SOUT samtools: 1.8-3 bioconda
[May 7 13:44:48] SOUT xz: 5.2.3-0 conda-forge
[May 7 13:44:48] SOUT zlib: 1.2.11-0 conda-forge
[May 7 13:44:48] SOUT
[May 7 13:44:49] SOUT Preparing transaction: ...working... done
[May 7 13:45:27] SOUT Verifying transaction: ...working... done
[May 7 13:45:31] SOUT Executing transaction: ...working... done
[May 7 13:45:39] DEBU Container [a9b07ae5f8ab step-57cca42fca] completed with exit code [0] as expected
[May 7 13:45:39] DEBU Container [a9b07ae5f8ab step-57cca42fca] removed
[May 7 13:45:39] STEP Wrap [build/dist] as [quay.io/local/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:23592e4ad15ca2acfca18facab87a1ce22c49da1-0]
[May 7 13:45:39] DEBU Creating container [step-76c9e42fa8]
[May 7 13:45:40] DEBU Created container [1980ef5d00d8 step-76c9e42fa8], starting it
[May 7 13:45:43] DEBU Container [1980ef5d00d8 step-76c9e42fa8] started, waiting for completion
[May 7 13:45:43] SERR [May 7 12:39:05] DEBU Creating container [step-075a575127]
[May 7 13:45:43] SERR [May 7 12:39:06] DEBU Packing succeeded
[May 7 13:45:46] DEBU Container [1980ef5d00d8 step-76c9e42fa8] completed with exit code [0] as expected
[May 7 13:45:46] DEBU Container [1980ef5d00d8 step-76c9e42fa8] removed
[May 7 13:45:46] INFO Invoke Task [singularity]
[May 7 13:45:46] WARN Invoked Stepless Task [singularity]
[May 7 13:45:46] INFO Invoke Task [cleanup]
[May 7 13:45:46] STEP Run image [continuumio/miniconda:latest] with command [[rm -rf /data/dist]]
[May 7 13:45:46] DEBU Creating container [step-b415da648f]
[May 7 13:45:47] DEBU Created container [ef00bf3e47b2 step-b415da648f], starting it
[May 7 13:45:51] DEBU Container [ef00bf3e47b2 step-b415da648f] started, waiting for completion
[May 7 13:45:51] DEBU Container [ef00bf3e47b2 step-b415da648f] completed with exit code [0] as expected
[May 7 13:45:51] DEBU Container [ef00bf3e47b2 step-b415da648f] removed
[May 7 13:45:51] INFO Invoke Task [test]
[May 7 13:45:51] STEP Run image [quay.io/local/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:23592e4ad15ca2acfca18facab87a1ce22c49da1-0] with command [[true]]
[May 7 13:45:51] DEBU Creating container [step-ecdec2a9ae]
[May 7 13:45:52] DEBU Created container [6b6b50be0c47 step-ecdec2a9ae], starting it
[May 7 13:45:56] DEBU Container [6b6b50be0c47 step-ecdec2a9ae] started, waiting for completion
[May 7 13:45:56] DEBU Container [6b6b50be0c47 step-ecdec2a9ae] completed with exit code [0] as expected
[May 7 13:45:56] DEBU Container [6b6b50be0c47 step-ecdec2a9ae] removed
```

**But right after this, when it tries to look up the container again:**

galaxy.tools.deps.containers INFO 2018-05-07 13:45:56,694 Checking with container resolver [BuildDockerContainerResolver[namespace=local]] found description [None]

and just stops looking up after calling BuildDockerContainerResolver only and moves on to resolving the dependency using conda (mulled v1):

galaxy.jobs.command_factory INFO 2018-05-07 13:45:56,754 Built script [/galaxy_base/scratch_prod/database/jwd/000/150/tool_script.sh] for tool command [[ "$CONDA_DEFAULT_ENV" = "/galaxy_base/src_prod/miniconda/envs/mulled-v1-9c20228f2237a66548a4779ba5e7f4b6fa6f4965c2871a6ac8c96b254401744b" ] ||

shouldn't it be trying to resolve again using MulledDockerContainerResolver and the rest of the resolvers?
At least that is what it does for dependencies whose container is found. E.g FastQC:

galaxy.tools.deps.containers INFO 2018-05-07 14:45:15,679 Checking with container resolver [ExplicitContainerResolver[]] found description [None]
docker images
galaxy.tools.deps.containers INFO 2018-05-07 14:45:15,735 Checking with container resolver [CachedMulledDockerContainerResolver[namespace=biocontainers]] found description [None]
requests.packages.urllib3.connectionpool INFO 2018-05-07 14:45:15,745 Starting new HTTPS connection (1): quay.io
requests.packages.urllib3.connectionpool DEBUG 2018-05-07 14:45:16,280 "GET /api/v1/repository/biocontainers/fastqc HTTP/1.1" 200 3685
galaxy.tools.deps.containers INFO 2018-05-07 14:45:16,283 Checking with container resolver [MulledDockerContainerResolver[namespace=biocontainers]] found description [ContainerDescription[identifier=quay.io/biocontainers/fastqc:0.11.7--pl5.22.0_2,type=docker]]

If I run docker images then the created image is listed. I can also run something using it manually, so the creation seems to be working ok. I'm not sure how the 'registry' works for this new image created locally but could that also be the issue? although the other resolvers are not showing the search None results.

As for configuration, I'm running it on Galaxy 17.09,
enable_beta_mulled_containers is set to True and containers_resolvers_config_file = None

Thanks in advance,
Ignacio

Contributor guide

Open the contributing guide

Research direction

Start in galaxy/tools/deps/containers.py and inspect the resolver sequence around BuildDockerContainerResolver and MulledDockerContainerResolver. Reproduce the multi-dependency case using the reported Galaxy 17.09 configuration and galaxy/tools/deps/mulled/invfile.lua. Done means a locally created involucro image is considered during the subsequent dependency lookup rather than falling through to conda.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.