common-workflow-language / common-workflow-language/cwltool
Change in behaviour when dealing with input file formats for tools
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
@khillion and I are currently working on a workflow that involves multiple CWL tools that were developed in different places. The two first steps are [mafft](https://github.com/C3BI-pasteur-fr/workflows/blob/master/tools/mafft.cwl) and then [noisy](https://github.com/C3BI-pasteur-fr/workflows/blob/master/tools/noisy.cwl), but we have a problem in the most recent versions of cwltool: the pipeline runs mafft fine, but then crashes at the time of running noisy, with the message
```
WorkflowException: Missing required 'format' for File {'checksum': u'sha1$a79f5fd26c5c5c4d9477b7e2cfaa22747b8f9b41', 'basename': u'mafft_alignment.fa', 'nameext': u'.fa', 'nameroot': u'mafft_alignment', 'http://commonwl.org/cwltool#generation': 0, 'location': 'toilfs:Z/B/tmpUn7zOI-x-mafft_alignment.fa', 'class': u'File', 'size': 5458L}
```
We believe this is caused by recent format checking modifications: the mafft cwl tool does not include format information in the output, but the noisy cwl tool specifies it). As a result, cwltool in its most recent versions refuses to accept the alignment file as an input of cwltool for noisy because the file object has no 'format' property. I haven't tracked much these developments, so I'm not sure if there were decisions taken, but if our interpretation is correct a side effect of this behaviour is that you absolutely need to specify formats for input files now when the tool specifies it, and i think this rule could be relaxed for the sake of usability.
We can reproduce the two behaviours easily (accepting and then refusing the format). The version where the absence of format is accepted is `cwltool==1.0.20180326152342`, and the one where it is refused is `cwltool==1.0.20180403145700`.
The "problem" could come from this modification: https://github.com/common-workflow-language/cwltool/commit/4c74d06385f0135a0859b94f4607d4c44f0b2b7d#diff-bfbf952b5c2585de423b12dcf6f43938R79
I would argue that adding a way to ignore format compatibility checking when there is no existing format specified in the input file would be a good thing. Any opinion on the matter?
Contributor guide
Assessment
This issue has not been assessed yet.