common-workflow-language / common-workflow-language/cwltool
tmp Dir exists already
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
I see a stochastic error for certain jobs when kicking off many jobs at the same time (~100 or so).
## Expected Behavior
`tmp dir` should be created once and subsequently checked for existence.
## Actual Behavior
```
OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'
```
## Command issued
```
toil-cwl-runner \
--logFile /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/log/cwltoil.log \
--jobStore file:///ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/tmp//jobstore-e18d4790-1ab8-11e9-bf33-70106fb1697e \
--batchSystem lsf \
--workDir /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/tmp/ \
--outdir /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV \
--writeLogs /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/log \
--logLevel INFO \
--stats \
--cleanWorkDir onSuccess \
--defaultDisk 10G \
--preserve-environment PATH PYTHONPATH TOIL_LSF_ARGS \
--retryCount 2 \
--no-container \
--defaultMem 10G \
--disableCaching \
--maxLogFileSize 20000000 \
/home/johnsoni/pipeline_0.1.0-DEV/ACCESS-Pipeline/workflows/subworkflows/snps_and_indels.cwl inputs.yaml
```
Note that the values that I am supplying for the `--workDir` and `--outdir` parameters are not in agreement with the `tmp` directory that is causing the error.
This mysterious `tmp` directory shows up in the current working directory from which the command was submitted.
I am using toil, and this directory has appeared since upgrading from 3.15 to 3.18.
However it seems like an issue with cwltool.
## Workflow Code
The pipeline I'm running is quite large, but I believe this issue is instead caused by submitting a large number of jobs concurrently.
I'm not familiar with all of the particular ways in which temporary directories are created by `cwltool`, and it seems there are several different ways of specifying the tmp dir location (`TMPDIR` environment variable, `--tmp-dir-prefix`, `--tmp-outdir-prefix`).
But my best guess is that there may be a race condition when creating the tmp dir because of this line:
https://github.com/common-workflow-language/cwltool/blob/1.0.20180819175200/cwltool/job.py#L381
```
if not os.path.exists(self.tmpdir):
os.makedirs(self.tmpdir)
```
Which will error if two jobs reach the same point at the same time without synchronization (similarly for any other places where this tmp dir is going to be created).
## Full Traceback
```pytb
WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'file:///home/johnsoni/pipeline_0.1.0-DEV/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java w/n/jobhgbJto
WARNING:toil.leader:w/n/jobhgbJto INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:w/n/jobhgbJto INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty.
WARNING:toil.leader:w/n/jobhgbJto Got workflow error
WARNING:toil.leader:w/n/jobhgbJto Traceback (most recent call last):
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:w/n/jobhgbJto job.run(runtime_context)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 378, in run
WARNING:toil.leader:w/n/jobhgbJto os.makedirs(self.tmpdir)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:w/n/jobhgbJto mkdir(name, mode)
WARNING:toil.leader:w/n/jobhgbJto OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'
WARNING:toil.leader:w/n/jobhgbJto ERROR:cwltool:Got workflow error
WARNING:toil.leader:w/n/jobhgbJto Traceback (most recent call last):
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:w/n/jobhgbJto job.run(runtime_context)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 378, in run
WARNING:toil.leader:w/n/jobhgbJto os.makedirs(self.tmpdir)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:w/n/jobhgbJto mkdir(name, mode)
WARNING:toil.leader:w/n/jobhgbJto OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'
```
## Your Environment
* cwltool version:
`1.0.20180820141117`
* Toil version:
`3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty`
Contributor guide
Assessment
This issue has not been assessed yet.