ch-run error: can't mkdir

Bug report

On HPC running nextflow with charliecloud.enabled=true submitting the pipeline call using bash -c 'nohup nextflow run ...'.

Observed error: ch-run[xxxxx]: error: can't mkdir: <path>: File exists (ch_misc.c:412 17)

Expected behavior and actual behavior

As observed with previous runs, the expected behavior is that reading and writing files during nextflow processes that execute inside charliecloud container should be no problem.

However the actual behavior shows multiple, independent processes observing the aforementioned error in some cases (e.g. 3/24 processes fail).
Furthermore, the part of the error message may be the path to a sample folder or a nextflow working dir.

Steps to reproduce the problem

On the HPC I am working on, run

bash -c 'nohup ~/<sample_group>_runner.sh > yyyymmdd_<sampe_group>.out &'

With the script containing the nextflow call:

nextflow run roland-rad-lab/MoCaSeq -r human-pipeline-nextflow-2  -profile charliecloud,slurm -work-dir /scratch/${PROJECT_ID}/${USER}/test/work --output_base ${PROJECT_DIR}/output --genome_build.human GRCh38.p12 --custom_config_version mocaseq --custom_config_base https://raw.githubusercontent.com/roland-rad-lab/MoCaSeq/human-pipeline-nextflow-2/conf --input ${INPUT_TABLE}

This results in following error:

ch-run[15383]: error: can't mkdir: <HOME>/images/<CONTAINER_NAME>/scratch/<PROJECT_ID>/<USER>/test/work/e3: File exists (ch_misc.c:412 17)

Program output

Here the respective part of .nextflow.log:

May-11 04:28:40.491 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=HUMAN_WGS:HMM_COPY:hmm_copy_wig_tumor (PCSI_0357_Pa_P_5262); work-dir=/scratch/${PROJECT_ID}/${USER}/test/work/08/cfd630734734679d51939159cc6d56
  error [nextflow.exception.ProcessFailedException]: Process `HUMAN_WGS:HMM_COPY:hmm_copy_wig_tumor (PCSI_0357_Pa_P_5262)` terminated with an error exit status (1)
May-11 04:28:40.571 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'HUMAN_WGS:HMM_COPY:hmm_copy_wig_tumor (PCSI_0357_Pa_P_5262)'

Caused by:
  Process `HUMAN_WGS:HMM_COPY:hmm_copy_wig_tumor (PCSI_0357_Pa_P_5262)` terminated with an error exit status (1)

Command executed:

  #!/usr/bin/env bash
  
  /opt/hmmcopy_utils/bin/readCounter -w 20000 -q20 -c 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,X,Y PCSI_0357_Pa_P_5262.Tumor.bam > PCSI_0357_Pa_P_5262.Tumor.20000.wig

Command exit status:
  1

Command output:
  (empty)

Command error:
  ch-run[56618]: error: can't mkdir: ${HOME}/images/<CONTAINER_NAME>/${PROJECT_DIR}/input/GRCh38.p12_bam/batch02/PCSI_0357_Pa_P_5262: File exists (ch_misc.c:412 17)

Work dir:
  /scratch/${PROJECT_ID}/${USER}/test/work/08/cfd630734734679d51939159cc6d56

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

The process script itself seems not to be involved to the error.

Environment

• Nextflow version: 23.04.1
• Java version: openjdk 11.0.13 2021-10-19
• Operating system: SUSE Linux Enterprise Server 15 SP1
• Bash version: 4.4.23(1)-release (x86_64-suse-linux-gnu)

Additional context

The pipeline is executed using SLURM.
I don't expect the ch_misc.c to be the error source, rather the binding of directories to the containers or a similar issue to #3367.
Is @phue still charliecloud master for nextflow?

Start by comparing the failing .nextflow.log entries with the generated .command.sh in the reported work directory, then trace the Nextflow Charliecloud and SLURM container-binding path for the conflicting mkdir. Done means identifying the binding or directory-creation cause, reproducing it reliably, and documenting or testing the corrected behavior.