nextflow-io / nextflow-io/nextflow
Caching using file on GitHub LFS
Open
Nobody has claimed this yet.
stale
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
We're using GitHub LFS to store some custom panels of normals for using ClairSTO. But we're having a caching issues, even though the link to the file on GitHub LFS works and doesn't seem to change, it generates a different hash in Nextflow each time leading to it not caching a chunk of the pipeline.
We've shasum265'd the files downloaded from GitHub on two separate runs, so they are definitely identical. If we use these links to our GitHub LFS files :
gnomad = "https://github.com/IntGenomicsLab/test-datasets/raw/refs/heads/main/ClairSTO-pon/final_gnomad.vcf.gz"
dbsnp = "https://github.com/IntGenomicsLab/test-datasets/raw/refs/heads/main/ClairSTO-pon/final_dbsnp.vcf.gz"
onekgenomes = "https://github.com/IntGenomicsLab/test-datasets/raw/refs/heads/main/ClairSTO-pon/final_1kgenomes.vcf.gz"
colors = "https://github.com/IntGenomicsLab/test-datasets/raw/refs/heads/main/ClairSTO-pon/final_colors.vcf.gz"
It causes caching problems as you can see comparing the hash dumps of two different runs
Log output
c9421a4641aef52f3b40e95a9c598fd4 [java.lang.String] .singularity/docker.io-hkubal-clairs-to-v0.4.0.img
22a9e5e0e97f0475e310f04c56b1393a [java.lang.String] meta
e49c1f5dc3789047917afffe71f20f42 [java.util.LinkedHashMap] [id:sample3, paired_data:false, platform:ont, sex:female, fiber:n, clair3_model:[], clairS_model:[], clairSTO_model:[], type:tumor, basecall_model:dna_r10.4.1_e8.2_400bps_sup@v5.0.0, kinetics:]
7a581e26b8a25430a95d80c42543225a [java.lang.String] tumor_bam
a36c09baabcd17f5b8269caebf27d74b [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam, storePath:/lustre1/scratch/364/vsc36452/work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam, stageName:sample3_mapped.bam)]
3b2001f3b337c50fb53aaec02b72e306 [java.lang.String] tumor_bai
e40da50167703cf107c48179c4cb19bd [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam.bai, storePath:/lustre1/scratch/364/vsc36452/work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam.bai, stageName:sample3_mapped.bam.bai)]
8c1adf6921bcf0ae92cbbf0ca7872591 [java.lang.String] model
99c1de48ea0cdf221b0502023add3d72 [java.lang.String] ont_r10_dorado_sup_5khz_ssrs
409b6322c96c5aed2e508e57cc521d5d [java.lang.String] meta2
00000000000000000000000000000000 [java.util.LinkedHashMap] [:]
fec83c41a7c57f821d28e6b995ac3b35 [java.lang.String] reference
8d751aa0d05300bbb7e19e161688100a [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/33/356e0da11c25bd5d3b7ace74594eef/GRCh38_chr19.fasta, storePath:/lustre1/scratch/364/vsc36452/work/33/356e0da11c25bd5d3b7ace74594eef/GRCh38_chr19.fasta, stageName:GRCh38_chr19.fasta)]
1948639f984981cfc51dfe462c11ba82 [java.lang.String] meta3
00000000000000000000000000000000 [java.util.LinkedHashMap] [:]
105e03d70ef5c389e573ff1c323ba2e5 [java.lang.String] index
ad60eb9ce956bab222d68c80e25e4212 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/01/d8a3ba664b218bf88467f7c0c47257/GRCh38_chr19.fasta.fai, storePath:/lustre1/scratch/364/vsc36452/work/01/d8a3ba664b218bf88467f7c0c47257/GRCh38_chr19.fasta.fai, stageName:GRCh38_chr19.fasta.fai)]
24efac282bd038f214d79f80d3511666 [java.lang.String] dbSNP
747965ed69c0fae5dd13fadd0896bdf1 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/fe/2c13a46b233466d4062c889d1e21e8/final_dbsnp.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/fe/2c13a46b233466d4062c889d1e21e8/final_dbsnp.vcf.gz, stageName:final_dbsnp.vcf.gz)]
12b5dfbd0bf1a478a13dc9bc5a9164c4 [java.lang.String] colors
ea9c14badb2f0fa03598a275c8c18fe6 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/f4/4b4276db70a2eb63841c066621e1c0/final_colors.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/f4/4b4276db70a2eb63841c066621e1c0/final_colors.vcf.gz, stageName:final_colors.vcf.gz)]
d5df2b145eb7b8356dbf50b1b1f242f8 [java.lang.String] onekgenomes
bdd62a02f3b8af0173e8addc9efcdb60 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/b0/e826a1fc1fefeee280beb9e426160f/final_1kgenomes.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/b0/e826a1fc1fefeee280beb9e426160f/final_1kgenomes.vcf.gz, stageName:final_1kgenomes.vcf.gz)]
24361f5846b1f2d7b3c01a6bd245ba60 [java.lang.String] gnomad
f4415eaa8cd2a4c42f1a159e38c5f3a9 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/17/99a8d883b22369988333ebf77b986b/final_gnomad.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/17/99a8d883b22369988333ebf77b986b/final_gnomad.vcf.gz, stageName:final_gnomad.vcf.gz)]
4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $
16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true
492c55335962ee2a44dda7e348856948 [java.util.HashMap$EntrySet] [task.ext.args=null, task.ext.prefix=null, workflow.containerEngine=singularity]
Log output
f52f3b40e95a9c598fd4 [java.lang.String] .singularity/docker.io-hkubal-clairs-to-v0.4.0.img
22a9e5e0e97f0475e310f04c56b1393a [java.lang.String] meta
e49c1f5dc3789047917afffe71f20f42 [java.util.LinkedHashMap] [id:sample3, paired_data:false, platform:ont, sex:female, fiber:n, clair3_model:[], clairS_model:[], clairSTO_model:[], type:tumor, basecall_model:dna_r10.4.1_e8.2_400bps_sup@v5.0.0, kinetics:]
7a581e26b8a25430a95d80c42543225a [java.lang.String] tumor_bam
a36c09baabcd17f5b8269caebf27d74b [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam, storePath:/lustre1/scratch/364/vsc36452/work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam, stageName:sample3_mapped.bam)]
3b2001f3b337c50fb53aaec02b72e306 [java.lang.String] tumor_bai
e40da50167703cf107c48179c4cb19bd [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam.bai, storePath:/lustre1/scratch/364/vsc36452/work/37/443074a27c1a57495f49d050e3be62/sample3_mapped.bam.bai, stageName:sample3_mapped.bam.bai)]
8c1adf6921bcf0ae92cbbf0ca7872591 [java.lang.String] model
99c1de48ea0cdf221b0502023add3d72 [java.lang.String] ont_r10_dorado_sup_5khz_ssrs
409b6322c96c5aed2e508e57cc521d5d [java.lang.String] meta2
00000000000000000000000000000000 [java.util.LinkedHashMap] [:]
fec83c41a7c57f821d28e6b995ac3b35 [java.lang.String] reference
8d751aa0d05300bbb7e19e161688100a [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/33/356e0da11c25bd5d3b7ace74594eef/GRCh38_chr19.fasta, storePath:work/33/356e0da11c25bd5d3b7ace74594eef/GRCh38_chr19.fasta, stageName:GRCh38_chr19.fasta)]
1948639f984981cfc51dfe462c11ba82 [java.lang.String] meta3
00000000000000000000000000000000 [java.util.LinkedHashMap] [:]
105e03d70ef5c389e573ff1c323ba2e5 [java.lang.String] index
ad60eb9ce956bab222d68c80e25e4212 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/01/d8a3ba664b218bf88467f7c0c47257/GRCh38_chr19.fasta.fai, storePath:/lustre1/scratch/364/vsc36452/work/01/d8a3ba664b218bf88467f7c0c47257/GRCh38_chr19.fasta.fai, stageName:GRCh38_chr19.fasta.fai)]
24efac282bd038f214d79f80d3511666 [java.lang.String] dbSNP
747965ed69c0fae5dd13fadd0896bdf1 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/fe/2c13a46b233466d4062c889d1e21e8/final_dbsnp.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/fe/2c13a46b233466d4062c889d1e21e8/final_dbsnp.vcf.gz, stageName:final_dbsnp.vcf.gz)]
12b5dfbd0bf1a478a13dc9bc5a9164c4 [java.lang.String] colors
ea9c14badb2f0fa03598a275c8c18fe6 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/f4/4b4276db70a2eb63841c066621e1c0/final_colors.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/f4/4b4276db70a2eb63841c066621e1c0/final_colors.vcf.gz, stageName:final_colors.vcf.gz)]
d5df2b145eb7b8356dbf50b1b1f242f8 [java.lang.String] onekgenomes
bdd62a02f3b8af0173e8addc9efcdb60 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/b0/e826a1fc1fefeee280beb9e426160f/final_1kgenomes.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/b0/e826a1fc1fefeee280beb9e426160f/final_1kgenomes.vcf.gz, stageName:final_1kgenomes.vcf.gz)]
24361f5846b1f2d7b3c01a6bd245ba60 [java.lang.String] gnomad
f4415eaa8cd2a4c42f1a159e38c5f3a9 [nextflow.util.ArrayBag] [FileHolder(sourceObj:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/17/99a8d883b22369988333ebf77b986b/final_gnomad.vcf.gz, storePath:work/stage-aab0ea9c-1309-47bb-ae52-276d12fb13ff/17/99a8d883b22369988333ebf77b986b/final_gnomad.vcf.gz, stageName:final_gnomad.vcf.gz)]
4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $
16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true
492c55335962ee2a44dda7e348856948 [java.util.HashMap$EntrySet] [task.ext.args=null, task.ext.prefix=null, workflow.containerEngine=singularity]
Whereas
gnomad = "http://www.bio8.cs.hku.hk/clairs-to/databases/gnomad.r2.1.af-ge-0.001.sites.vcf.gz"
dbsnp = "http://www.bio8.cs.hku.hk/clairs-to/databases/dbsnp.b138.non-somatic.sites.vcf.gz"
onekgenomes = "http://www.bio8.cs.hku.hk/clairs-to/databases/1000g-pon.sites.vcf.gz"
colors = "http://www.bio8.cs.hku.hk/clairs-to/databases/CoLoRSdb.GRCh38.v1.1.0.deepvariant.glnexus.af-ge-0.001.vcf.gz"
With no differences in handling the variables, causes no caching problems. This is odd to me since both seem to be consistent https links and the two variables are handled identically
Is this an known problem with GitHub LFS or a Nextflow bug?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Nextflow issue details, comparing the GitHub LFS URLs and the two supplied hash dumps with the working HTTP URLs. Reproduce the two runs and determine whether the differing cache hashes originate in GitHub LFS handling or Nextflow; done means the cause is identified and caching behavior is consistently explained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, groovy
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100