Charliecloud simultaneous pull error

Bug report

Expected behavior and actual behavior

Running an nf-core pipeline would download the charliecloud images on-demand.

When multiple different processes are scheduled at the same time, requiring different images, there is an error message and nextflow stops (error shown below).

Steps to reproduce the problem

When there are no local cached charliecloud images, the following command can reproduce the error:

nextflow run nf-core/raredisease -r dev  --outdir raredisease --genome GRCh38 --analysis_type wgs  -profile test,charliecloud

Program output

Error executing process > 'NFCORE_RAREDISEASE:RAREDISEASE:CHECK_INPUT:SAMPLESHEET_CHECK (1)'

Caused by:
  Charliecloud failed to pull image
  command: ch-image pull -s /data/ephemeral/charlie quay.io/biocontainers/python:3.9--1 > /dev/null
  status : 1
  message:
    error: storage directory is already in use
    hint: concurrent instances of ch-image cannot share the same storage directory

Environment

• Nextflow version: 23.01.0-edge
• Java version: openjdk 11.0.13
• Operating system: Linux
• Bash version: 5.1.16(1)-release
• Charliecloud version: 0.30 (from conda)

Additional context

It's possible to work around the problem by manually pulling all the required images before running nextflow.

Start by running the provided nf-core/raredisease command with no cached Charliecloud images and review the image-pull handling around the reported storage-directory error. Reproduce concurrent pulls for different images, then verify that the workflow completes without requiring manual pre-pulling.