common-workflow-language / common-workflow-language/cwltool
possible regression with --parallel
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
I have a fairly complicated workflow which succeeds with `--parallel` when using cwltool 1.0.20180809224403 but fails with latest master. It succeeds without `--parallel` on both versions.
The workflow is in https://github.com/wtsi-hgi/arvados-pipelines, branch syndip-eval, commit bf860ab, running as:
```
cwltool --parallel ~/arvados-pipelines/cwl/workflows/gatk-4.0.0.0-library-cram-to-gvcfs.cwl input.yaml
```
The failure looks like this:
```
....
[step combine_haplotype_index_2] start
[step combine_haplotype_index_2] start
[step combine_haplotype_index_2] start
Exception on step 'combine_haplotype_index'
Cannot make scatter job: Invalid job input record:
../arvados-pipelines/cwl/expression-tools/combine_files.cwl:19:3: Missing required input parameter 'main_file'
Workflow cannot make any more progress.
Final process status is permanentFail
```
I git bisected it down to this commit, with what looks like it might be an unintended indent change at "if job.outdir:"
https://github.com/common-workflow-language/cwltool/commit/91e9e9f655fb149a2d3344a69883e147e94b2f50#diff-e4a1a3aa00120b3bcef5dfd8b66f7633R299
but reverting that indent change has not made the workflow succeed.
I'm not familiar with the internals of cwltool. What else can I do to try and diagnose why this workflow doesn't work with `--parallel` on latest master? Is there something wrong with the workflow?
Thanks,
Dave
## Expected Behavior
Workflow should work with `--parallel` on latest cwltool.
## Actual Behavior
Workflow fails with `--parallel`, when it used to work previously (version 1.0.20180809224403)
## Workflow Code
From the repo https://github.com/wtsi-hgi/arvados-pipelines, branch syndip-eval, commit
bf860ab. In workflows/gatk-4.0.0.0-haplotypecaller.cwl in "combine_haplotype_index"
```
Exception on step 'combine_haplotype_index'
Cannot make scatter job: Invalid job input record:
../arvados-pipelines/cwl/expression-tools/combine_files.cwl:19:3: Missing required input parameter 'main_file'
Workflow cannot make any more progress.
Final process status is permanentFail
```
## Full Traceback
No traceback is given.
## Your Environment
* cwltool version: 1.0.20181203095715
Contributor guide
Assessment
This issue has not been assessed yet.