common-workflow-language / common-workflow-language/cwltool

Problem with when statements in the workflow: nonetype input for scatter

Open
#1,800 3 comments 0 reactions 1 assignee Claimed by @tetron View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I have 2 following steps in CWL workflow that are executed conditionally (using when statement) and output from one of them is input for the second one, and second one contains scatter. There is a fragment of code here:

```
steps:
list_fastqs_from_dirs_with_ngs_data:
run: ../shared_tools/collections2files/collections2files.cwl
when: $(inputs.run_step === true)
in:
dirs: dirs_with_ngs_data
run_step: run_sort_fastq
out: [files]

sort_fastq:
run: ./sort_fastq/sort_fastq.cwl
when: $(inputs.run_sort_fastq === true)
in:
fastq_file: list_fastqs_from_dirs_with_ngs_data/files
run_sort_fastq: run_sort_fastq
scatter: fastq_file
out: [sorted_fastq]
```

When I run the pipeline and `run_sort_fastq === true` pipeline is completed.
But when `run_sort_fastq === false` pipeline fails, because scatter is empty:

```
ERROR Unhandled exception
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/lib/python3.9/site-packages/cwltool/workflow_job.py", line 730, in try_make_job
emptyscatter = [
File "/usr/local/Caskroom/miniconda/base/lib/python3.9/site-packages/cwltool/workflow_job.py", line 731, in
shortname(s) for s in scatter if len(cast(Sized, inputobj[s])) == 0
TypeError: object of type 'NoneType' has no len()
```
It should not matter that scatter is empty, because `sort_fastq` step is not executed.
I fixed the problem, using dummy input 'blank_input' in `sort_fastq` with the same input type than `list_fastqs_from_dirs_with_ngs_data/files` that is always not empty. The code with solution looks like that:

```
fastq_file:
source: [list_fastqs_from_dirs_with_ngs_data/files, blank_input]
pickValue: first_non_null
```
and works.

Exactly the same problem also exists in arvados-cwl-runner, which I described here: https://cwl.discourse.group/t/problem-with-when-statements-in-the-workflow-with-nonetype-input-for-scatter/749/1

cwltool version: 3.1.20230127121939

Could you look into it please?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.