nf-core / nf-core/tools

Use RefGenie for pipeline template `genomes.conf` config

Open
#1,084 4 comments 0 reactions 2 assignees View on GitHub

@ErikDanielsson is already working on this.

Since Jun 17, 2022.

automation template
Dominant language
Python
Stars
322
Forks
255
Avg merge
2d 3h
Merged PRs (30d)
5

Description

Split from #592 - specifically https://github.com/nf-core/tools/issues/592#issuecomment-804419834


I wanted to announce an update that would allow using refgenie for unarchived cloud assets directly. You first need a refgenie digest for the genome of interest, which you can get at an endpoint like this: http://rg.databio.org/genomes/genome_digest/hg38.

Then, you can use that with the assets/file_path endpoint to return either an http or s3 URL to the file of interest. For example:

you can also use individual seek keys, just like the CLI, to get individual items within an asset:
http://rg.databio.org/assets/file_path/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/chrom_sizes?remoteClass=s3

One way to auto-generate a config file is to use the new refgenie populate function.

you would create a template like:

params {
  // illumina iGenomes reference file paths
  genomes {
    'GRCh37' {
      fasta       = "refgenie://hg38/fasta"
      bwa         = "refgenie://hg38/bwa_index"
      bowtie2     = "refgenie://hg38/bowtie2_index"
      }
    }
}

then you just run some flavor of refgenie populate file.tpl and you'd get the above, using the then-current URIs


So the idea would be to make a copy of the template conf/igenomes.conf file (need to rename to just genomes.conf) that is called conf/igenomes.conf.tpl. Then build that with the asset identifiers as described above.

Once that's done, we run refgenie populate on that file to create the conf/genomes.conf file with the absolute assets. This file should never be directly edited.

To ensure this, we should have a CI test that regenerates the file and checks that it matches what is committed to the repo (eg. git diff doesn't return anything).

Many / most of the reference genome assets that we currently have will be missing for now I guess, so this ties in with issue https://github.com/nf-core/tools/issues/1086 to create those. But we can leave them commented out and in a branch for now until they are ready.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.