nextflow-io / nextflow-io/nextflow
Native process resubmitting even though inputs are the same.
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
I have a native process that resubmits itself on -resume even though the inputs don't change, but only if I've cleaned previous executions.
Expected behavior and actual behavior
I expect the process to be cached. Nextflow only uses a cached version if I don't clean previous runs.
Steps to reproduce the problem
https://github.com/NBISweden/Earth-Biogenome-Project-pilot
nextflow run main.nf -profile docker,test -resume
nextflow clean -f -before $( nextflow log -q | tail -n 1 ) && find work/ -type d -empty -delete
nextflow run main.nf -profile docker,test -resume
If I don't clean, then the process uses a cached hit.
Program output
I posted the question in the community forum initially:
https://community.seqera.io/t/why-would-native-processes-exec-be-resubmitted-even-though-the-hash-inputs-dont-change/2259
hashes from run1
Jul-03 14:03:28.903 [Actor Thread 5] INFO nextflow.processor.TaskProcessor - [ASSEMBLY_REPORT:TOL_SEARCH (1)] cache hash: 96e3c94a530ae88fb7647046ec705b3e; mode: STANDARD; entries:
76ea2021cc55a873477f77be45e8a1fc [java.util.UUID] 8e358b4c-7f8d-4421-8742-0a77a3f8c4d6
bdf226854332f21139e1ddac952e22ba [java.lang.String] ASSEMBLY_REPORT:TOL_SEARCH
f13fc1b539c963cfd9f7a5eda57f24b3 [java.lang.String] def args = task.ext.args ?: ''
def response = new URL("https://id.tol.sanger.ac.uk/api/v2/species?taxonomyId=$taxid").text
def lazy_json = new groovy.json.JsonSlurper().parseText(response)
json = [
tol_id: lazy_json.species[0]['tolIds'][0]?.tolId?: "No ToL ID",
species: lazy_json.species[0]['scientificName'],
class: lazy_json.species[0]['taxaClass'],
order: lazy_json.species[0]['order'],
]
16cb95a8f5388c13899a651695911162 [java.lang.String] taxid
dec56b9915c1138e7cd4677f712b5833 [java.lang.Integer] 7227
4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $
16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true
eab359affdb9334848cc02803d7db724 [java.util.HashMap$EntrySet] [task.ext.args=null]
Jul-03 14:03:28.922 [Task submitter] INFO nextflow.Session - [c1/d3eb41] Submitted process > ASSEMBLY_REPORT:TOL_SEARCH (Taxid: 7227)
and hashes from run2
Jul-03 14:04:33.476 [Actor Thread 2] INFO nextflow.processor.TaskProcessor - [ASSEMBLY_REPORT:TOL_SEARCH (1)] cache hash: 96e3c94a530ae88fb7647046ec705b3e; mode: STANDARD; entries:
76ea2021cc55a873477f77be45e8a1fc [java.util.UUID] 8e358b4c-7f8d-4421-8742-0a77a3f8c4d6
bdf226854332f21139e1ddac952e22ba [java.lang.String] ASSEMBLY_REPORT:TOL_SEARCH
f13fc1b539c963cfd9f7a5eda57f24b3 [java.lang.String] def args = task.ext.args ?: ''
def response = new URL("https://id.tol.sanger.ac.uk/api/v2/species?taxonomyId=$taxid").text
def lazy_json = new groovy.json.JsonSlurper().parseText(response)
json = [
tol_id: lazy_json.species[0]['tolIds'][0]?.tolId?: "No ToL ID",
species: lazy_json.species[0]['scientificName'],
class: lazy_json.species[0]['taxaClass'],
order: lazy_json.species[0]['order'],
]
16cb95a8f5388c13899a651695911162 [java.lang.String] taxid
dec56b9915c1138e7cd4677f712b5833 [java.lang.Integer] 7227
4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $
16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true
eab359affdb9334848cc02803d7db724 [java.util.HashMap$EntrySet] [task.ext.args=null]
Jul-03 14:04:33.497 [Task submitter] INFO nextflow.Session - [c1/d3eb41] Submitted process > ASSEMBLY_REPORT:TOL_SEARCH (Taxid: 7227)
Environment
- Nextflow version: 25.04.4
- Java version: [?]
- Operating system: [macOS, Linux, etc]
- Bash version: (use the command
$SHELL --version)
Additional context
(Add any other context about the problem here)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked reproduction's main.nf and run the two nextflow commands around nextflow clean, then compare the cache hashes and work/ directory state shown in the report. Read the cache and native-process handling reached by that reproduction. Done means an unchanged native process is restored from cache after prior executions are cleaned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100