common-workflow-language / common-workflow-language/cwltool

Error regarding cachedir ceation for job without 'success' status but still with a cachedir

Open
#1,267 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
After a crash I was expecting to go on with my analysis using cache dir

## Actual Behavior

When there is a cache dir with `.status` set to `success` it does behave as expected. However when there is the `permanentFail` status, cwl tries to create a dir with the `os.makedirs(jobcache)` python instruction. Then a `FileExistsError` raises because the cachedir corresponding to the permanentfail already exists.

## Workflow Code
I used the option `--cachedir /media/hdd_storage/tmp/BATCH_NAME_._pipeline-name_._1585223540/d9106db4979617eaf77c35c474411088`

## Full Traceback
```pytb
[2020-03-27 09:05:18] INFO [job fastq_screen_20] Output of job will be cached in /media/hdd_storage/tmp/BATCH_NAME_._pipeline-name_._1585223540/d9106db4979617eaf77c35c474411088
[2020-03-27 09:05:18] ERROR Unexpected exception
Traceback (most recent call last):
File "/home/user/.local/lib/python3.6/site-packages/cwltool/workflow.py", line 1043, in job
runtimeContext,
File "/home/user/.local/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 530, in job
os.makedirs(jobcache)
File "/usr/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/media/hdd_storage/tmp/BATCH_NAME_._pipeline-name_._1585223540/d9106db4979617eaf77c35c474411088'
[2020-03-27 09:05:18] ERROR Cannot make scatter job: [Errno 17] File exists: '/media/hdd_storage/tmp/BATCH_NAME_._pipeline-name_._1585223540/d9106db4979617eaf77c35c474411088'
[2020-03-27 09:05:18] DEBUG
Traceback (most recent call last):
File "/home/user/.local/lib/python3.6/site-packages/cwltool/workflow.py", line 1043, in job
runtimeContext,
File "/home/user/.local/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 530, in job
os.makedirs(jobcache)
File "/usr/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/media/hdd_storage/tmp/BATCH_NAME_._pipeline-name_._1585223540/d9106db4979617eaf77c35c474411088'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/user/.local/lib/python3.6/site-packages/cwltool/workflow.py", line 1124, in parallel_st
eps
for j in step:
File "/home/user/.local/lib/python3.6/site-packages/cwltool/workflow.py", line 265, in job
for j in self.step.job(joborder, output_callback, runtimeContext):
File "/home/user/.local/lib/python3.6/site-packages/cwltool/workflow.py", line 1051, in job
raise WorkflowException(str(exc)) from exc
cwltool.errors.WorkflowException: [Errno 17] File exists: '/media/hdd_storage/tmp/BATCH_NAME_._pipeline-name_._1585223540/d9106db4979617eaf77c35c474411088'
```

## Your Environment
* cwltool version:
`cwltool 3.0.20200324120055`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.