nextflow-io / nextflow-io/nextflow

Native process resubmitting even though inputs are the same.

Open
#6,246 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage/investigate
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

I have a native process that resubmits itself on -resume even though the inputs don't change, but only if I've cleaned previous executions.

Expected behavior and actual behavior

I expect the process to be cached. Nextflow only uses a cached version if I don't clean previous runs.

Steps to reproduce the problem

https://github.com/NBISweden/Earth-Biogenome-Project-pilot

nextflow run main.nf -profile docker,test -resume
nextflow clean -f -before $( nextflow log -q | tail -n 1 ) && find work/ -type d -empty -delete
nextflow run main.nf -profile docker,test -resume

If I don't clean, then the process uses a cached hit.

Program output

I posted the question in the community forum initially:
https://community.seqera.io/t/why-would-native-processes-exec-be-resubmitted-even-though-the-hash-inputs-dont-change/2259

hashes from run1

Jul-03 14:03:28.903 [Actor Thread 5] INFO  nextflow.processor.TaskProcessor - [ASSEMBLY_REPORT:TOL_SEARCH (1)] cache hash: 96e3c94a530ae88fb7647046ec705b3e; mode: STANDARD; entries: 
  76ea2021cc55a873477f77be45e8a1fc [java.util.UUID] 8e358b4c-7f8d-4421-8742-0a77a3f8c4d6 
  bdf226854332f21139e1ddac952e22ba [java.lang.String] ASSEMBLY_REPORT:TOL_SEARCH 
  f13fc1b539c963cfd9f7a5eda57f24b3 [java.lang.String]     def args = task.ext.args ?: ''
    def response = new URL("https://id.tol.sanger.ac.uk/api/v2/species?taxonomyId=$taxid").text
    def lazy_json = new groovy.json.JsonSlurper().parseText(response)
    json = [
        tol_id:  lazy_json.species[0]['tolIds'][0]?.tolId?: "No ToL ID",
        species: lazy_json.species[0]['scientificName'],
        class:   lazy_json.species[0]['taxaClass'],
        order:   lazy_json.species[0]['order'],
    ]
 
  16cb95a8f5388c13899a651695911162 [java.lang.String] taxid 
  dec56b9915c1138e7cd4677f712b5833 [java.lang.Integer] 7227 
  4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $ 
  16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true 
  eab359affdb9334848cc02803d7db724 [java.util.HashMap$EntrySet] [task.ext.args=null] 

Jul-03 14:03:28.922 [Task submitter] INFO  nextflow.Session - [c1/d3eb41] Submitted process > ASSEMBLY_REPORT:TOL_SEARCH (Taxid: 7227)

and hashes from run2

Jul-03 14:04:33.476 [Actor Thread 2] INFO  nextflow.processor.TaskProcessor - [ASSEMBLY_REPORT:TOL_SEARCH (1)] cache hash: 96e3c94a530ae88fb7647046ec705b3e; mode: STANDARD; entries: 
  76ea2021cc55a873477f77be45e8a1fc [java.util.UUID] 8e358b4c-7f8d-4421-8742-0a77a3f8c4d6 
  bdf226854332f21139e1ddac952e22ba [java.lang.String] ASSEMBLY_REPORT:TOL_SEARCH 
  f13fc1b539c963cfd9f7a5eda57f24b3 [java.lang.String]     def args = task.ext.args ?: ''
    def response = new URL("https://id.tol.sanger.ac.uk/api/v2/species?taxonomyId=$taxid").text
    def lazy_json = new groovy.json.JsonSlurper().parseText(response)
    json = [
        tol_id:  lazy_json.species[0]['tolIds'][0]?.tolId?: "No ToL ID",
        species: lazy_json.species[0]['scientificName'],
        class:   lazy_json.species[0]['taxaClass'],
        order:   lazy_json.species[0]['order'],
    ]
 
  16cb95a8f5388c13899a651695911162 [java.lang.String] taxid 
  dec56b9915c1138e7cd4677f712b5833 [java.lang.Integer] 7227 
  4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $ 
  16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true 
  eab359affdb9334848cc02803d7db724 [java.util.HashMap$EntrySet] [task.ext.args=null] 

Jul-03 14:04:33.497 [Task submitter] INFO  nextflow.Session - [c1/d3eb41] Submitted process > ASSEMBLY_REPORT:TOL_SEARCH (Taxid: 7227)
Environment
  • Nextflow version: 25.04.4
  • Java version: [?]
  • Operating system: [macOS, Linux, etc]
  • Bash version: (use the command $SHELL --version)
Additional context

(Add any other context about the problem here)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked reproduction's main.nf and run the two nextflow commands around nextflow clean, then compare the cache hashes and work/ directory state shown in the report. Read the cache and native-process handling reached by that reproduction. Done means an unchanged native process is restored from cache after prior executions are cleaned.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.