Use RefGenie for pipeline template `genomes.conf` config
@ErikDanielsson is already working on this.
Since Jun 17, 2022.
- Dominant language
- Python
- Stars
- 322
- Forks
- 255
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 5
Description
Split from #592 - specifically https://github.com/nf-core/tools/issues/592#issuecomment-804419834
I wanted to announce an update that would allow using refgenie for unarchived cloud assets directly. You first need a refgenie digest for the genome of interest, which you can get at an endpoint like this: http://rg.databio.org/genomes/genome_digest/hg38.
Then, you can use that with the assets/file_path endpoint to return either an http or s3 URL to the file of interest. For example:
- http://rg.databio.org/assets/file_path/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/fasta?remoteClass=s3
- http://rg.databio.org/assets/file_path/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/fasta?remoteClass=http
you can also use individual seek keys, just like the CLI, to get individual items within an asset:
http://rg.databio.org/assets/file_path/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/chrom_sizes?remoteClass=s3One way to auto-generate a config file is to use the new
refgenie populatefunction.you would create a template like:
params { // illumina iGenomes reference file paths genomes { 'GRCh37' { fasta = "refgenie://hg38/fasta" bwa = "refgenie://hg38/bwa_index" bowtie2 = "refgenie://hg38/bowtie2_index" } } }then you just run some flavor of refgenie populate
file.tpland you'd get the above, using the then-current URIs
So the idea would be to make a copy of the template conf/igenomes.conf file (need to rename to just genomes.conf) that is called conf/igenomes.conf.tpl. Then build that with the asset identifiers as described above.
Once that's done, we run refgenie populate on that file to create the conf/genomes.conf file with the absolute assets. This file should never be directly edited.
To ensure this, we should have a CI test that regenerates the file and checks that it matches what is committed to the repo (eg. git diff doesn't return anything).
Many / most of the reference genome assets that we currently have will be missing for now I guess, so this ties in with issue https://github.com/nf-core/tools/issues/1086 to create those. But we can leave them commented out and in a branch for now until they are ready.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.