common-workflow-language / common-workflow-language/cwltool
cwltool pack creates invalid json when sub-attributes are named the same
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
## Expected Behavior
Either `cwltool --validate` should fail on a cwl or an error is raised on pack, not just a warning.
## Actual Behavior
`cwltool --pack /path/to/workflow.cwl` creates an invalid json when an output attribute and a step attribute have the same name
## Workflow Code
Note that the outputs attribute is named 'samtools_flagstat' as is the single step.
### Workflow
```
#!/usr/bin/env cwl-runner
cwlVersion: v1.1
class: Workflow
# Metadata
id: WTS QC Workflow for ISL Showcase
label: wts-qc
doc: This is the WTS QC Workflow
# Requirements
requirements:
MultipleInputFeatureRequirement: {}
StepInputExpressionRequirement: {}
SubworkflowFeatureRequirement: {}
# Inputs
inputs:
# Sample name used to determine file output name
sample_name: string
sample_bam_filepath:
type: File
format: http://edamontology.org/format_2572
streamable: true
secondaryFiles:
- .bai
# Outputs
outputs:
samtools_flagstat:
type: File
outputSource: samtools_flagstat/flagstat_output
# Steps
steps:
samtools_flagstat:
in:
sample_name:
source: sample_name
sample_bam_filepath:
source: sample_bam_filepath
out: [flagstat_output]
run: "samtools-flagstat.ctl.cwl"
```
### Commandline Tool
```
cwlVersion: v1.0
class: CommandLineTool
# Metadata
id: samtools_flagstat
label: samtools-flagstat
doc: This is samtools flagstat in place as a proof of principle.
# Requirements
requirements:
- class: ResourceRequirement
ramMin: 2000
coresMin: 1
- class: DockerRequirement
dockerPull: 'quay.io/biocontainers/samtools:1.10--h9402c20_2'
- class: InlineJavascriptRequirement
# Base command
baseCommand: [ "samtools", "flagstat" ]
# Inputs
inputs:
# Sample name used to determine file output name
sample_name:
type: string
label: Sample Name
doc: Name of the sample
# Bam file, must be in bam format and have a .bai suffix
sample_bam_filepath:
type: File
# Ensure it is of bam format
format: http://edamontology.org/format_2572
streamable: true
doc: Path to the bam file
secondaryFiles:
- '.bai'
threads:
type: int?
default: 1
inputBinding:
prefix: "-@"
separate: true
# Outputs
outputs:
flagstat_output:
type: stdout
# Samtools flagstat arguments
arguments: [ "$(inputs['sample_bam_filepath'].path)" ]
# Capture stdout with sample_name-flagstat.txt
stdout: "$(inputs.sample_name)-flagstat.txt"
# Ensure successful output
successCodes:
- 0
```
## Full Traceback
### Validate the commandline tool
```
$ cwltool --validate samtools-flagstat.ctl.cwl
samtools-flagstat.ctl.cwl is valid CWL.
```
### Validate the workflow
```
cwltool --validate samtools.flagstat.wfl.cwl
INFO /home/alexiswl/anaconda3/envs/cwl/bin/cwltool 3.0.20200324120055
INFO Resolved 'samtools.flagstat.wfl.cwl' to 'file:///a/.../samtools.flagstat.wfl.cwl'
samtools.flagstat.wfl.cwl:30:3: object id `samtools.flagstat.wfl.cwl#WTS QC Workflow for ISL Showcase/samtools_flagstat` previously defined
WARNING Workflow checker warning:
samtools.flagstat.wfl.cwl:22:5: Source 'sample_bam_filepath' of type "File" may be incompatible
samtools.flagstat.wfl.cwl:40:7: with sink 'sample_bam_filepath' of type "File"
samtools.flagstat.wfl.cwl is valid CWL.
```
### Pack the cwl workflow
```
$ cwltool --debug --pack samtools.flagstat.wfl.cwl > samtools.flagstat.wfl.cwl.json
# Produces the following to stderr
INFO /home/alexiswl/anaconda3/envs/cwl/bin/cwltool 3.0.20200324120055
INFO Resolved 'samtools.flagstat.wfl.cwl' to 'file:///a/...samtools.flagstat.wfl.cwl'
samtools.flagstat.wfl.cwl:28:3: object id `samtools.flagstat.wfl.cwl#WTS QC Workflow for ISL Showcase/samtools_flagstat` previously defined
samtools.flagstat.wfl.cwl:28:3: object id `samtools.flagstat.wfl.cwl#samtools_flagstat` previously defined
# But produces a non-zero exit code
$ echo "$?"
0
```
### Validate the packed cwl workflow
```pytb
$ cwltool --debug --validate samtools.flagstat.wfl.cwl.json
INFO /home/alexiswl/anaconda3/envs/cwl/bin/cwltool 3.0.20200324120055
INFO Resolved 'samtools.flagstat.wfl.cwl.json' to 'file:///a/.../samtools.flagstat.wfl.cwl.json'
ERROR Tool definition failed validation:
samtools.flagstat.wfl.cwl.json:73:9: checking object `samtools.flagstat.wfl.cwl.json#main`
samtools.flagstat.wfl.cwl.json:107:13: checking field `outputs`
samtools.flagstat.wfl.cwl.json:108:17: checking object
`samtools.flagstat.wfl.cwl.json#samtools_flagstat`
samtools.flagstat.wfl.cwl.json:110:21: Field `outputSource` references unknown identifier
`file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output`,
tried
file:///a/.../samtools.flagstat.wfl.cwl.json#samtools_flagstat/file:///a/.../samto
ols.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output,
file:///a/.../samtools.flagstat.wfl.cwl.json#file:///a/.../samtools.flagstat.wfl.c
wl.json#/samtools_flagstat/flagstat_output
samtools.flagstat.wfl.cwl.json:114:13: checking field `steps`
samtools.flagstat.wfl.cwl.json:115:17: checking object
`samtools.flagstat.wfl.cwl.json#samtools_flagstat`
samtools.flagstat.wfl.cwl.json:110:21: Field `outputSource` references unknown identifier
`file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output`,
tried
file:///a/.../samtools.flagstat.wfl.cwl.json#samtools_flagstat/file:///a/.../samto
ols.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output,
file:///a/.../samtools.flagstat.wfl.cwl.json#file:///a/.../samtools.flagstat.wfl.c
wl.json#/samtools_flagstat/flagstat_output
Traceback (most recent call last):
File "/home/alexiswl/anaconda3/envs/cwl/lib/python3.8/site-packages/cwltool-3.0.20200324120055-py3.8.egg/cwltool/main.py", line 912, in main
loadingContext, uri = resolve_and_validate_document(
File "/home/alexiswl/anaconda3/envs/cwl/lib/python3.8/site-packages/cwltool-3.0.20200324120055-py3.8.egg/cwltool/load_tool.py", line 336, in resolve_and_validate_document
document_loader.resolve_all(workflowobj, fileuri)
File "/home/alexiswl/anaconda3/envs/cwl/lib/python3.8/site-packages/schema_salad-5.0.20200302192450-py3.8.egg/schema_salad/ref_resolver.py", line 1192, in resolve_all
loader.validate_links(
File "/home/alexiswl/anaconda3/envs/cwl/lib/python3.8/site-packages/schema_salad-5.0.20200302192450-py3.8.egg/schema_salad/ref_resolver.py", line 1422, in validate_links
raise errors[0]
schema_salad.exceptions.ValidationException: samtools.flagstat.wfl.cwl.json:73:9: checking object `samtools.flagstat.wfl.cwl.json#main`
samtools.flagstat.wfl.cwl.json:107:13: checking field `outputs`
samtools.flagstat.wfl.cwl.json:108:17: checking object
`samtools.flagstat.wfl.cwl.json#samtools_flagstat`
samtools.flagstat.wfl.cwl.json:110:21: Field `outputSource` references unknown identifier
`file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output`,
tried
file:///a/.../samtools.flagstat.wfl.cwl.json#samtools_flagstat/file:///a/.../samto
ols.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output,
file:///a/.../samtools.flagstat.wfl.cwl.json#file:///a/.../samtools.flagstat.wfl.c
file:///a/.../samtools.flagstat.wfl.cwl.json#samtools_flagstat/file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output,
file:///a/.../samtools.flagstat.wfl.cwl.json#file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output
samtools.flagstat.wfl.cwl.json:114:13: checking field `steps`
samtools.flagstat.wfl.cwl.json:115:17: checking object
`samtools.flagstat.wfl.cwl.json#samtools_flagstat`
samtools.flagstat.wfl.cwl.json:110:21: Field `outputSource` references unknown identifier
`file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output`,
tried
file:///a/.../samtools.flagstat.wfl.cwl.json#samtools_flagstat/file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output,
file:///a/.../samtools.flagstat.wfl.cwl.json#file:///a/.../samtools.flagstat.wfl.cwl.json#/samtools_flagstat/flagstat_output
```
## Your Environment
* cwltool version:
Check using ``cwltool --version``
```
/home/alexiswl/anaconda3/envs/cwl/bin/cwltool 3.0.20200324120055
```
Contributor guide
Assessment
This issue has not been assessed yet.