common-workflow-language / common-workflow-language/cwltool

tmp Dir exists already

Open
#1,036 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I see a stochastic error for certain jobs when kicking off many jobs at the same time (~100 or so).

## Expected Behavior
`tmp dir` should be created once and subsequently checked for existence.

## Actual Behavior
```
OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'
```

## Command issued
```
toil-cwl-runner \
--logFile /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/log/cwltoil.log \
--jobStore file:///ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/tmp//jobstore-e18d4790-1ab8-11e9-bf33-70106fb1697e \
--batchSystem lsf \
--workDir /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/tmp/ \
--outdir /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV \
--writeLogs /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/log \
--logLevel INFO \
--stats \
--cleanWorkDir onSuccess \
--defaultDisk 10G \
--preserve-environment PATH PYTHONPATH TOIL_LSF_ARGS \
--retryCount 2 \
--no-container \
--defaultMem 10G \
--disableCaching \
--maxLogFileSize 20000000 \
/home/johnsoni/pipeline_0.1.0-DEV/ACCESS-Pipeline/workflows/subworkflows/snps_and_indels.cwl inputs.yaml
```

Note that the values that I am supplying for the `--workDir` and `--outdir` parameters are not in agreement with the `tmp` directory that is causing the error.

This mysterious `tmp` directory shows up in the current working directory from which the command was submitted.

I am using toil, and this directory has appeared since upgrading from 3.15 to 3.18.

However it seems like an issue with cwltool.

## Workflow Code
The pipeline I'm running is quite large, but I believe this issue is instead caused by submitting a large number of jobs concurrently.

I'm not familiar with all of the particular ways in which temporary directories are created by `cwltool`, and it seems there are several different ways of specifying the tmp dir location (`TMPDIR` environment variable, `--tmp-dir-prefix`, `--tmp-outdir-prefix`).

But my best guess is that there may be a race condition when creating the tmp dir because of this line:

https://github.com/common-workflow-language/cwltool/blob/1.0.20180819175200/cwltool/job.py#L381

```
if not os.path.exists(self.tmpdir):
os.makedirs(self.tmpdir)
```

Which will error if two jobs reach the same point at the same time without synchronization (similarly for any other places where this tmp dir is going to be created).

## Full Traceback
```pytb
WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'file:///home/johnsoni/pipeline_0.1.0-DEV/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java w/n/jobhgbJto
WARNING:toil.leader:w/n/jobhgbJto INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:w/n/jobhgbJto INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty.
WARNING:toil.leader:w/n/jobhgbJto Got workflow error
WARNING:toil.leader:w/n/jobhgbJto Traceback (most recent call last):
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:w/n/jobhgbJto job.run(runtime_context)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 378, in run
WARNING:toil.leader:w/n/jobhgbJto os.makedirs(self.tmpdir)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:w/n/jobhgbJto mkdir(name, mode)
WARNING:toil.leader:w/n/jobhgbJto OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'
WARNING:toil.leader:w/n/jobhgbJto ERROR:cwltool:Got workflow error
WARNING:toil.leader:w/n/jobhgbJto Traceback (most recent call last):
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:w/n/jobhgbJto job.run(runtime_context)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 378, in run
WARNING:toil.leader:w/n/jobhgbJto os.makedirs(self.tmpdir)
WARNING:toil.leader:w/n/jobhgbJto File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:w/n/jobhgbJto mkdir(name, mode)
WARNING:toil.leader:w/n/jobhgbJto OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'
```

## Your Environment
* cwltool version:
`1.0.20180820141117`
* Toil version:
`3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.