Clean up behavior silently broken if working directory is removed externally

Bug report

(Please follow this template replacing the text between parentheses with the requested information)

Expected behavior and actual behavior

Expected:

• nextflow clean -f -before $( nextflow log -q | tail -n 1 ) should not clean up cached results required in the last run
• nextflow clean should be robust against changes to work/ from external commands

Actual:

• If work/ has been manually deleted, all nextflow clean command later will erroneously delete cached results silently

Steps to reproduce the problem

Workflow main.nf:

workflow {
    TASK1(1)
    | TASK2
}

process TASK1 {
    input:
    val it

    script:
    """
    echo $it > nums.txt
    """

    output:
    path "nums.txt"
}

process TASK2 {
    //cache false
    input:
    path numfile

    script:
    """
    cat $numfile
    """

    output:
    stdout
}

Commands to reproduce:

rm -rf .nextflow* work/ -rf
nextflow run main.nf -resume
nextflow run main.nf -resume
rm -rf work/ # External removal of work/
nextflow run main.nf -resume
nextflow run main.nf -resume
nextflow clean -f -before $( nextflow log -q | tail -n 1 ) # Erroneous clean up
nextflow run main.nf -resume
nextflow clean -f -before $( nextflow log -q | tail -n 1 ) # Erroneous ever after

Program output

$ rm -rf .nextflow* work/ -rf
$
$
$ nextflow run main.nf -resume
N E X T F L O W  ~  version 23.10.1
WARN: It appears you have never run this project before -- Option `-resume` is ignored
Launching `main.nf` [berserk_lalande] DSL2 - revision: e917b3f26d
executor >  local (2)
[b7/90f502] process > TASK1 [100%] 1 of 1 ✔
[f0/04649c] process > TASK2 [100%] 1 of 1 ✔

$ nextflow run main.nf -resume
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [sad_woese] DSL2 - revision: e917b3f26d
[b7/90f502] process > TASK1 [100%] 1 of 1, cached: 1 ✔
[f0/04649c] process > TASK2 [100%] 1 of 1, cached: 1 ✔

$ rm -rf work/
$ nextflow run main.nf -resume
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [scruffy_raman] DSL2 - revision: e917b3f26d
executor >  local (2)
[b7/90f502] process > TASK1 [100%] 1 of 1 ✔
[5c/7fd47b] process > TASK2 [100%] 1 of 1 ✔

$ nextflow run main.nf -resume
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [gloomy_hodgkin] DSL2 - revision: e917b3f26d
[b7/90f502] process > TASK1 [100%] 1 of 1, cached: 1 ✔
[5c/7fd47b] process > TASK2 [100%] 1 of 1, cached: 1 ✔

$ nextflow clean -f -before $( nextflow log -q | tail -n 1 )
Removed /tmp/test-nf/work/b7/90f50242429d3424bbfcd8cc58afb5
Removed /tmp/test-nf/work/f0/04649c5649fca82635a49820d3ba02
$ ls /tmp/test-nf/work/b7/90f50242429d3424bbfcd8cc58afb5
ls: cannot access '/tmp/test-nf/work/b7/90f50242429d3424bbfcd8cc58afb5': No such file or directory
$ ls /tmp/test-nf/work/f0/04649c5649fca82635a49820d3ba02
ls: cannot access '/tmp/test-nf/work/f0/04649c5649fca82635a49820d3ba02': No such file or directory
$ nextflow run main.nf -resume
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [tender_gates] DSL2 - revision: e917b3f26d
executor >  local (2)
[b7/90f502] process > TASK1 [100%] 1 of 1 ✔
[65/cb4125] process > TASK2 [100%] 1 of 1 ✔

$ nextflow clean -f -before $( nextflow log -q | tail -n 1 )
Removed /tmp/test-nf/work/b7/90f50242429d3424bbfcd8cc58afb5
Removed /tmp/test-nf/work/5c/7fd47b06630d0efd1c86a3c4e20f06
$

Environment

• Nextflow version: 23.10.1
• Java version: openjdk version "17.0.9" 2023-10-17
• Operating system:Linux
• Bash version: 5.2.26(1)-release (x86_64-redhat-linux-gnu)

Additional context

Note that TASK1's generated hash collide. This probably causes the clean command to delete it every time, causing every resume thereafter to rerun everything. Removing a visible scratch directory is a pretty natural thing to do, so I would expect that nextflow can catch any desynchronization between .nextflow/ and work/, or at the very least emit an error instead of proceeding with a destructive operation in a corrupted state.

Reproduce the issue with the provided main.nf workflow and command sequence, especially removing work/ before running nextflow clean. Start by tracing how nextflow clean and -resume reconcile .nextflow/ state with work/; done means cleanup does not silently remove required cached results after external work/ removal, or reports the desynchronized state safely.