common-workflow-language / common-workflow-language/cwltool

cwltool fail when multiple jobs try to use the same conda env

Open
#2,164 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I tried to run workflow in a cluster using a shared conda installation.
I have created all conda envs required by the workflow.
Full command:

```
cwltool --disable-js-validation --skip-schemas --no-container --beta-conda-dependencies --beta-dependencies-directory /path/to/folder/with/_conda/ --rm-tmpdir --tmpdir-prefix=/eTMP/1559223537/ /download_quality_control_single.cwl sra_download_SRR10043607.yml
```

Running multiple jobs at the same time produce this error:
```
INFO [workflow ] start
INFO [workflow ] starting step fastq_dump
INFO [step fastq_dump] start
INFO [job fastq_dump] /export/TMP/1559223537/6jb6l69q$ fastq-dump \
--gzip \
--split-files \
SRR10043607 \
--defline-seq \
'@$ac.$si/$ri' \
--skip-technical
ERROR Exception while running job: Failed to get file lock for /path/to/folder/with/conda.
WARNING [job fastq_dump] completed permanentFail
ERROR [step fastq_dump] Output is missing expected field file:////download_quality_control_single.cwl#download_quality_control/fastq_dump/output
WARNING [step fastq_dump] completed permanentFail
INFO [workflow ] completed permanentFail
{
"fastq": null,
"fastqc_html": null,
"fastqc_zip": null
}WARNING Final process status is permanentFail
```

How can I make cwltool to use conda but not to lock the conda dir?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.