common-workflow-language / common-workflow-language/cwltool

CWL storing outputs into temp file and throwing error

Open
#969 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior

I have CWL script which is used to run an in house python script which takes in BAM files as input and gives out graphs. The script can be executed in parts that is in first part generate the graphs and in the second part merge the generated graphs. Currently if I run the python scripts on command line it creates the graphs and in the second run it searches in the output directory and merges the graph.
I also notice that CWL writes all output to temp strach directory and at the end writes the outputs to the defined output directory. And so it automatically never search in the working directory. And If I give a absolute path then it is throwing error message.

## Actual Behavior
The script should run and say `- All result files already exist.` and continue with merging the graph
as the parameter given is `merge_graphs`

## The CWL workflow code
```

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow

doc: "Workflow Components: Increment -> Spladder -> PeptidePackage"

requirements:
- class: ScatterFeatureRequirement

inputs:
spladder_gtf:
type: File
spladder_bams:
type: File[]
secondaryFiles:
.bai
spladder_outDir:
type: Directory
spladder_confidence:
type: int
spladder_merge_graphs:
type: string
spladder_phase2:
type: string
spladder_alt:
type: string
spladder_RL:
type: int
spladder_validate:
type: string

outputs:
incre_output:
type: File
outputSource: increment_count/incre_output
spladder_out1:
type: Directory
outputSource: spladder/spladder_out_dir1
spladder_out2:
type: Directory
outputSource: spladder/spladder_out_dir2

steps:
spladder:
run: spladder1.cwl
in:
spladder_gtf: spladder_gtf
spladder_bams: spladder_bams
spladder_outDir: spladder_outDir
spladder_confidence: spladder_confidence
spladder_merge_graphs: spladder_merge_graphs
spladder_phase2: spladder_phase2
spladder_alt: spladder_alt
spladder_RL: spladder_RL
spladder_validate: spladder_validate
out: [spladder_out_dir1, spladder_out_dir2, spladderFile]
```

The spladder command line.

```
cwlVersion: v1.0
class: CommandLineTool
doc: Spladder

baseCommand: [python2.7, /cluster/home/aalva/spladder/python/spladder.py]

requirements:
- class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement
listing:
- entry: "$({class: 'Directory', listing: []})"
entryname: $(inputs.spladder_outDir)
writable: true

inputs:
spladder_gtf:
type: File
inputBinding:
position: 3
prefix: -a
spladder_bam:
type: File
inputBinding:
position: 1
prefix: -b
secondaryFiles: .bai
spladder_outDir:
type: Directory
inputBinding:
position: 2
prefix: -o
spladder_phase2:
type: string
inputBinding:
position: 6
prefix: -T
spladder_merge_graphs:
type: string
inputBinding:
position: 5
prefix: -M
spladder_primary_alignment:
type: string
inputBinding:
position: 10
prefix: -P
spladder_confidence:
type: int
inputBinding:
position: 4
prefix: -c
spladder_alt:
type: string
inputBinding:
position: 7
prefix: -t
spladder_validate:
type: string
inputBinding:
position: 8
prefix: -V
spladder_RL:
type: int
inputBinding:
position: 9
prefix: -n

outputs:
spladder_out:
type: Directory
outputBinding:
glob: $(inputs.spladder_outDir)/spladder

```

## Full Traceback
```
Traceback (most recent call last):
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/executors.py", line 167, in run_jobs
job.run(runtime_context)
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/job.py", line 418, in run
symLink=True, secret_store=runtimeContext.secret_store)
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/process.py", line 257, in stageFiles
os.makedirs(p.target, 0o0755)
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/Alignment/spladder_out/'
Workflow error, try again with --debug for more information:
[Errno 17] File exists: '/Alignment/spladder_out/'
```
The YML file
```
spladder_gtf:
class: File
path: /usage_examples/gencode.v19.annotation.hs37d5_chr.spladder.gtf
spladder_outDir:/Alignment/spladder_out/
spladder_out_dir1: /spladder_out1
spladder_out_dir2: /spladder_out2
spladder_bams: [
{class: File, path: /Alignment/C3N-02289_10_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/C3N-02289_4_5_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /cluster/work/grlab/projects/alva_temp/Alignment/C3N-02671_08_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/C3N-02671_10_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/EOC04_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/GY150806_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /JM-2_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/Me275_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/Me290_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/MM170111_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/MM909362_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/MM909432_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/RCC14_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/RCC16_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/T1015A_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/T1185B_L1Aligned.sortedByCoord.out.bam}
]
spladder_confidence: 2
spladder_merge_graphs: merge_graphs
spladder_alt: alt_3prime
spladder_RL: 100
spladder_phase2: y
spladder_primary_alignment: y
```
## Your Environment
* cwltool version:
Check using ``cwltool --version v1.0``

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.