aryarm / aryarm/as_analysis

improve dependency installation on cloud

Open
#56 0 comments 0 reactions 0 assignees View on GitHub
enhancement gcp
Dominant language
R
Stars
10
Forks
9
PR merge metrics
No merged PRs in 30d

Description

the google life sciences api creates a new snakemake container to run each of our jobs
so our dependencies must be installed anew by snakemake every time it runs a job, since each job is run in a new, empty container
this process may add substantially to the runtime of some of our instances and the cost of using them

we have two options:
1) separate our dependencies into different env files to install only the necessary dependencies for each job
2) create a custom snakemake container with our dependencies already installed in it and force snakemake to use our container instead of its own

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by locating the Snakemake configuration for Google Life Sciences jobs and its dependency environment or container settings. Compare the two proposed approaches, then consider the work done when jobs no longer reinstall unnecessary dependencies or use a custom container with the required dependencies preinstalled.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud
Domain
bioinformatics, cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.