common-workflow-language / common-workflow-language/cwltool

possible regression with --parallel

Open
#999 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I have a fairly complicated workflow which succeeds with `--parallel` when using cwltool 1.0.20180809224403 but fails with latest master. It succeeds without `--parallel` on both versions.

The workflow is in https://github.com/wtsi-hgi/arvados-pipelines, branch syndip-eval, commit bf860ab, running as:

```
cwltool --parallel ~/arvados-pipelines/cwl/workflows/gatk-4.0.0.0-library-cram-to-gvcfs.cwl input.yaml
```

The failure looks like this:

```
....
[step combine_haplotype_index_2] start
[step combine_haplotype_index_2] start
[step combine_haplotype_index_2] start
Exception on step 'combine_haplotype_index'
Cannot make scatter job: Invalid job input record:
../arvados-pipelines/cwl/expression-tools/combine_files.cwl:19:3: Missing required input parameter 'main_file'
Workflow cannot make any more progress.
Final process status is permanentFail
```

I git bisected it down to this commit, with what looks like it might be an unintended indent change at "if job.outdir:"

https://github.com/common-workflow-language/cwltool/commit/91e9e9f655fb149a2d3344a69883e147e94b2f50#diff-e4a1a3aa00120b3bcef5dfd8b66f7633R299

but reverting that indent change has not made the workflow succeed.

I'm not familiar with the internals of cwltool. What else can I do to try and diagnose why this workflow doesn't work with `--parallel` on latest master? Is there something wrong with the workflow?

Thanks,
Dave

## Expected Behavior
Workflow should work with `--parallel` on latest cwltool.

## Actual Behavior
Workflow fails with `--parallel`, when it used to work previously (version 1.0.20180809224403)

## Workflow Code
From the repo https://github.com/wtsi-hgi/arvados-pipelines, branch syndip-eval, commit
bf860ab. In workflows/gatk-4.0.0.0-haplotypecaller.cwl in "combine_haplotype_index"
```
Exception on step 'combine_haplotype_index'
Cannot make scatter job: Invalid job input record:
../arvados-pipelines/cwl/expression-tools/combine_files.cwl:19:3: Missing required input parameter 'main_file'
Workflow cannot make any more progress.
Final process status is permanentFail
```

## Full Traceback

No traceback is given.

## Your Environment
* cwltool version: 1.0.20181203095715

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.