common-workflow-language / common-workflow-language/cwltool
workflow with parallel steps and --beta-conda-dependencies failed due to many conda install at the same time
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
the workflow should install the Conda packages before submitting the parallel steps.
## Actual Behavior
Running a workflow with `--no-container --beta-conda-dependencies ` and the SoftwareRequirement pointing to Conda packages fail due to multiple installations at the same time.
```
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/_conda/lib/python3.7/site-packages/conda/core/package_cache_data.py", line 661, in _execute_actions
extract_axn.execute(progress_update_extract_axn)
File "/home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/_conda/lib/python3.7/site-packages/conda/core/path_actions.py", line 1194, in execute
backoff_rename(self.target_full_path, self.hold_path)
File "/home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/_conda/lib/python3.7/site-packages/conda/gateways/disk/update.py", line 88, in backoff_rename
exp_backoff_fn(rename, source_path, destination_path, force)
File "/home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/_conda/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 25, in exp_backoff_fn
return fn(*args, **kwargs)
File "/home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/_conda/lib/python3.7/site-packages/conda/gateways/disk/update.py", line 57, in rename
os.rename(source_path, destination_path)
OSError: [Errno 39] Directory not empty: '/home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/_conda/pkgs/perl-file-path-2.16-pl526_0' -> '/home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/_conda/pkgs/perl-file-path-2.16-pl526_0.c~'
```
## Workflow Code
```
https://github.com/ncbi/cwl-ngs-workflows-cbb/blob/master/workflows/sra/download_quality_control.cwl
```
## Command:
```
cwltool --no-container --beta-conda-dependencies --beta-dependencies-directory /home/veraalva/rnaseq-sra-paired/bin/cwltool_deps/ --parallel --on-error continue --rm-tmpdir --tmp-outdir-prefix=/home/veraalva/rnaseq-sra-paired/tmp/ --tmpdir-prefix=/home/veraalva/rnaseq-sra-paired/tmp/ https://raw.githubusercontent.com/ncbi/cwl-ngs-workflows-cbb/master/workflows/sra/download_quality_control.cwl --threads 4 --ncbi_config /home/veraalva/.ncbi --split-files -X 10000000 --accession SRR2126784 --accession SRR2126785 --accession SRR2126786 --accession SRR2126787 --accession SRR3383790 --accession SRR3383791 --accession SRR3383792 --accession SRR3383794 --accession SRR3383796 --accession SRR3383797 --accession SRR3383798 --accession SRR2126788 --accession SRR2126789 --accession SRR2126790 --accession SRR2126791 --accession SRR2126792 --accession SRR2126793 --accession SRR2126794 --accession SRR2126795 --accession SRR2126796
```
Contributor guide
Assessment
This issue has not been assessed yet.