common-workflow-language / common-workflow-language/cwltool

ValidationException error due to incompatiblility

Open
#1,041 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behaviour

Hello All,

I have a CWL script I need to run within toil-cal runner. The script is simple Fastq >quality control> alignment > read quantification. I am using linkMerge:merge_flattened to collect the inputs from previous steps in the workflow.

## Actual Behaviour

The CWL is throwing consistently the following error,
`schema_salad.validate.ValidationException: ` and it complains the incompatibility

## Workflow Code
The CWL for Toil logistics
`pipeline_parallel.cwl`
````
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow

requirements:
- class: ScatterFeatureRequirement
- class: SubworkflowFeatureRequirement
- class: InlineJavascriptRequirement

inputs:
reads1:
type: File[]
reads2:
type: File[]
sample_names:
type: string[]
genomeDir:
type: Directory
sjdbGTFfile:
type: File
annotation:
type: File
exp_out:
type: string[]
#genome:
#type: File
#outSAMattrRGline: string

outputs:
bam_dir:
type: Directory
outputSource: collect/bam_dir
fastqc_dir:
type: Directory
outputSource: collect/fastqc_dir
count_exp:
type: Directory
outputSource: collect/count_exp
bam_index:
type: Directory
outputSource: collect/bam_index
#spladder_dir:
#type: Directory
#outputSource: collect/spladder_dir

steps:
pipeline_workflow:
run: expression_count_pipeline.cwl
scatter: [reads1, reads2, sample_name, exp_out]
#Runs concurrently all the inputs in the array
scatterMethod: dotproduct
in:
reads1: reads1
reads2: reads2
sample_name: sample_names
sample_names: sample_names
exp_out: sample_names
genomeDir: genomeDir
sjdbGTFfile: sjdbGTFfile
annotation: annotation
out: [fastqc_out, fastqc_html, alignment_out, index_out, expression_out]
collect:
in:
fastqc_files:
source: [pipeline_workflow/fastqc_out, pipeline_workflow/fastqc_html]
#Collects inputs from source or previous steps in the workflow and run
linkMerge: merge_flattened
bam_files:
source: [pipeline_workflow/alignment_out]
linkMerge: merge_flattened
bam_index:
source: [pipeline_workflow/index_out]
linkMerge: merge_flattened
count_files:
source: [pipeline_workflow/expression_out]
linkMerge: merge_flattened
#splice_files:
#source: [pipeline_workflow/spladder_out]
#linkMerge: merge_nested
out: [bam_dir, fastqc_dir, count_exp, bam_index]
run:
class: ExpressionTool
id: "collect_step"
inputs:
fastqc_files:
type:
type: array
items:
type: array
items: File
bam_files: File[]
bam_index: File[]
count_files: File[]
outputs:
bam_dir: Directory
fastqc_dir: Directory
bam_index: Directory
count_exp: Directory
expression: |
${
return {
"fastqc_dir": {
"class": "Directory",
"basename": "fastqc",
"listing": [].concat.apply([], inputs.fastqc_files)
},
"bam_dir": {
"class": "Directory",
"basename": "bams",
"listing": inputs.bam_files
},
"count_exp": {
"class": "Directory",
"basename": "Expression",
"listing": inputs.count_exp
}
};
}
```

The YML file
pipeline.yml

```
reads1: # array of type "File"
- class: File
path: /home/fastqs/JM-2_L1_R1_001_ugGztrZAOnWJ.fastq.gz
- class: File
path: /home/fastqs/Me275_L1_R1_001_KSb7V0YJOooV.fastq.gz
reads2: # array of type "File"
- class: File
path /home/fastqs/JM-2_L1_R2_001_8G1u1kKn2Bnq.fastq.gz
- class: File
path: /home/fastqs/Me275_L1_R2_001_CdXkX3p8jzei.fastq.gz
sample_names: # array of type "File"
- JM-2
- Me275
genomeDir:
class: Directory
path:/genome/hg19_hs37d5.overhang100_STAR
sjdbGTFfile:
class: File
path:/annotation/gencode.v28lift37.annotation.gtf
fastqc_check_script:
class: File
path: /rnaseq/fastqc_check.sh
annotation:
class: File
path: /scripts/gencode.v19.annotation.hs37d5_chr.gtf

```

## Full Traceback
```pytb
DEBUG:toil.lib.bioio:Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
DEBUG:toil.lib.bioio:Logging to file 'toil.log'.
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/cluster/work/grlab/projects/alva_temp/Toil_poc/my_base_dir1/jobstore'.
DEBUG:toil.jobStores.abstractJobStore:The workflow ID is: '0929e6a4-2440-4696-a63e-7a1d58f9761a'
INFO:cwltool:Resolved 'pipeline_main_parallel.cwl' to 'file:///cluster/work/grlab/projects/alva_temp/Toil_poc/pipeline_main_parallel.cwl'
pipeline_main_parallel.cwl:92:9: object id `pipeline_main_parallel.cwl#collect/collect_step/bam_index` previously defined
WARNING:salad:pipeline_main_parallel.cwl:92:9: object id `pipeline_main_parallel.cwl#collect/collect_step/bam_index` previously defined
pipeline_main_parallel.cwl:92:9: object id `pipeline_main_parallel.cwl#collect/collect_step/bam_index` previously defined
WARNING:salad:pipeline_main_parallel.cwl:92:9: object id `pipeline_main_parallel.cwl#collect/collect_step/bam_index` previously defined
Traceback (most recent call last):
File "/cluster/home/aalva/software/anaconda/envs/toil/bin/toil-cwl-runner", line 11, in
sys.exit(main())
File "/cluster/home/aalva/software/anaconda/envs/toil/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1203, in main
loading_context)
File "/cluster/home/aalva/software/anaconda/envs/toil/lib/python2.7/site-packages/cwltool/load_tool.py", line 346, in make_tool
tool = loadingContext.construct_tool_object(processobj, loadingContext)
File "/cluster/home/aalva/software/anaconda/envs/toil/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 286, in toil_make_tool
return cwltool.workflow.default_make_tool(toolpath_object, loading_context)
File "/cluster/home/aalva/software/anaconda/envs/toil/lib/python2.7/site-packages/cwltool/workflow.py", line 54, in default_make_tool
return Workflow(toolpath_object, loadingContext)
File "/cluster/home/aalva/software/anaconda/envs/toil/lib/python2.7/site-packages/cwltool/workflow.py", line 556, in __init__
static_checker(workflow_inputs, workflow_outputs, step_inputs, step_outputs, param_to_step)
File "/cluster/home/aalva/software/anaconda/envs/toil/lib/python2.7/site-packages/cwltool/checker.py", line 214, in static_checker
raise validate.ValidationException(all_exception_msg)
schema_salad.validate.ValidationException: pipeline_main_parallel.cwl:16:3: Source 'sample_names' of type {"items": "string", "type": "array"}
is incompatible
pipeline_main_parallel.cwl:57:7: with sink 'exp_out' of type {"items": {"items": "string",
"type": "array"}, "type": "array"}
```

## Your Environment
CWL version 1.0

I ran the CWL scripts as,

`toil-cwl-runner --stats --clusterStats --retryCount=0 --batchSystem=lsf --disableCaching --tmpdir-prefix ${TMP_DIR} --tmp-outdir-prefix ${TMP_OUT_DIR} --workDir ${WORK_DIR} --cleanWorkDir=never --clean=never --outdir ${OUT_DIR} --logDebug --logFile ${LOG_FILE} --jobStore ${JOB_STORE} pipeline_parallel.cwl pipeline.yml`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.