common-workflow-language / common-workflow-language/cwltool
CWL storing outputs into temp file and throwing error
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
I have CWL script which is used to run an in house python script which takes in BAM files as input and gives out graphs. The script can be executed in parts that is in first part generate the graphs and in the second part merge the generated graphs. Currently if I run the python scripts on command line it creates the graphs and in the second run it searches in the output directory and merges the graph.
I also notice that CWL writes all output to temp strach directory and at the end writes the outputs to the defined output directory. And so it automatically never search in the working directory. And If I give a absolute path then it is throwing error message.
## Actual Behavior
The script should run and say `- All result files already exist.` and continue with merging the graph
as the parameter given is `merge_graphs`
## The CWL workflow code
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
doc: "Workflow Components: Increment -> Spladder -> PeptidePackage"
requirements:
- class: ScatterFeatureRequirement
inputs:
spladder_gtf:
type: File
spladder_bams:
type: File[]
secondaryFiles:
.bai
spladder_outDir:
type: Directory
spladder_confidence:
type: int
spladder_merge_graphs:
type: string
spladder_phase2:
type: string
spladder_alt:
type: string
spladder_RL:
type: int
spladder_validate:
type: string
outputs:
incre_output:
type: File
outputSource: increment_count/incre_output
spladder_out1:
type: Directory
outputSource: spladder/spladder_out_dir1
spladder_out2:
type: Directory
outputSource: spladder/spladder_out_dir2
steps:
spladder:
run: spladder1.cwl
in:
spladder_gtf: spladder_gtf
spladder_bams: spladder_bams
spladder_outDir: spladder_outDir
spladder_confidence: spladder_confidence
spladder_merge_graphs: spladder_merge_graphs
spladder_phase2: spladder_phase2
spladder_alt: spladder_alt
spladder_RL: spladder_RL
spladder_validate: spladder_validate
out: [spladder_out_dir1, spladder_out_dir2, spladderFile]
```
The spladder command line.
```
cwlVersion: v1.0
class: CommandLineTool
doc: Spladder
baseCommand: [python2.7, /cluster/home/aalva/spladder/python/spladder.py]
requirements:
- class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement
listing:
- entry: "$({class: 'Directory', listing: []})"
entryname: $(inputs.spladder_outDir)
writable: true
inputs:
spladder_gtf:
type: File
inputBinding:
position: 3
prefix: -a
spladder_bam:
type: File
inputBinding:
position: 1
prefix: -b
secondaryFiles: .bai
spladder_outDir:
type: Directory
inputBinding:
position: 2
prefix: -o
spladder_phase2:
type: string
inputBinding:
position: 6
prefix: -T
spladder_merge_graphs:
type: string
inputBinding:
position: 5
prefix: -M
spladder_primary_alignment:
type: string
inputBinding:
position: 10
prefix: -P
spladder_confidence:
type: int
inputBinding:
position: 4
prefix: -c
spladder_alt:
type: string
inputBinding:
position: 7
prefix: -t
spladder_validate:
type: string
inputBinding:
position: 8
prefix: -V
spladder_RL:
type: int
inputBinding:
position: 9
prefix: -n
outputs:
spladder_out:
type: Directory
outputBinding:
glob: $(inputs.spladder_outDir)/spladder
```
## Full Traceback
```
Traceback (most recent call last):
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/executors.py", line 167, in run_jobs
job.run(runtime_context)
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/job.py", line 418, in run
symLink=True, secret_store=runtimeContext.secret_store)
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/process.py", line 257, in stageFiles
os.makedirs(p.target, 0o0755)
File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/Alignment/spladder_out/'
Workflow error, try again with --debug for more information:
[Errno 17] File exists: '/Alignment/spladder_out/'
```
The YML file
```
spladder_gtf:
class: File
path: /usage_examples/gencode.v19.annotation.hs37d5_chr.spladder.gtf
spladder_outDir:/Alignment/spladder_out/
spladder_out_dir1: /spladder_out1
spladder_out_dir2: /spladder_out2
spladder_bams: [
{class: File, path: /Alignment/C3N-02289_10_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/C3N-02289_4_5_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /cluster/work/grlab/projects/alva_temp/Alignment/C3N-02671_08_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/C3N-02671_10_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/EOC04_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/GY150806_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /JM-2_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/Me275_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/Me290_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/MM170111_L1Aligned.sortedByCoord.out.bam},
{class: File, path:/Alignment/MM909362_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/MM909432_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/RCC14_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/RCC16_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/T1015A_L1Aligned.sortedByCoord.out.bam},
{class: File, path: /Alignment/T1185B_L1Aligned.sortedByCoord.out.bam}
]
spladder_confidence: 2
spladder_merge_graphs: merge_graphs
spladder_alt: alt_3prime
spladder_RL: 100
spladder_phase2: y
spladder_primary_alignment: y
```
## Your Environment
* cwltool version:
Check using ``cwltool --version v1.0``
Contributor guide
Assessment
This issue has not been assessed yet.