nf-core / nf-core/tools

RefGenie - add assets currently used from AWS-iGenomes

Open
#1,086 4 comments 0 reactions 2 assignees View on GitHub

@ErikDanielsson is already working on this.

Since Jun 17, 2022.

template
Dominant language
Python
Stars
322
Forks
255
Avg merge
2d 3h
Merged PRs (30d)
5

Description

Split from #592, related directly to #1084


We want to transition away from using AWS-iGenomes to using the central RefGenie server(s) to host reference genome assets for the nf-core pipelines. To do this, we need to make sure that everything we currently have is available on a AWS refgenie server somewhere. Once all assets are mirrored we can do a clean swap in the config.

Going from igenomes.config, here is a check-list:

Summary of genomes
  • GRCh37
  • GRCh38
  • GRCm38
  • TAIR10
  • EB2
  • UMD3.1
  • WBcel235
  • CanFam3.1
  • GRCz10
  • BDGP6
  • EquCab2
  • EB1
  • Galgal4
  • Gm01
  • Mmul_1
  • IRGSP-1.0
  • CHIMP2.1.4
  • Rnor_6.0
  • R64-1-1
  • EF2
  • Sbi1
  • Sscrofa10.2
  • AGPv3
  • hg38
  • hg19
  • mm10
  • bosTau8
  • ce10
  • canFam3
  • danRer10
  • dm6
  • equCab2
  • galGal4
  • panTro4
  • rn6
  • sacCer3
  • susScr3
Detailed version with asset types
  • GRCh37
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • GRCh38
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
    • blacklist
  • GRCm38
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • TAIR10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • EB2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • UMD3.1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • WBcel235
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • CanFam3.1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • GRCz10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • BDGP6
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • EquCab2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • EB1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • Galgal4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • Gm01
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • Mmul_1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • IRGSP-1.0
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • CHIMP2.1.4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • Rnor_6.0
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • R64-1-1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • EF2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
  • Sbi1
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
  • Sscrofa10.2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • AGPv3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • hg38
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
    • blacklist
  • hg19
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • mm10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
    • blacklist
  • bosTau8
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • ce10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
    • macs_gsize
  • canFam3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • danRer10
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • dm6
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
    • macs_gsize
  • equCab2
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • galGal4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • panTro4
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name
  • rn6
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • mito_name
  • sacCer3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • readme
    • mito_name
    • macs_gsize
  • susScr3
    • fasta
    • bwa
    • bowtie2
    • star
    • bismark
    • gtf
    • bed12
    • readme
    • mito_name

From @nsheff:

For updating: We drive the server instances off a git repository, here: https://github.com/refgenie/refgenomes.databio.org

RIght now, it's semi-automated. Eventually we want it to be that you just update the repo with a PR, and when merged it will deploy automatically. For now, though, we have to build the things manually, but it's all scripted from that repository. The asses are all just annotated with a PEP. So, to add a genome you'd add it to this CSV file: https://github.com/refgenie/refgenomes.databio.org/blob/master/asset_pep/genome_descriptions.csv

Then you'd add whatever inputs are required to build the assets to this file: https://github.com/refgenie/refgenomes.databio.org/blob/master/asset_pep/recipe_inputs.csv

See also:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.