common-workflow-language / common-workflow-language/cwltool

Scattering on multiple fields with secondary files

Open
#1,208 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
CWLTool should scatter the files with their secondary files. I'm primarily using the `dotproduct`, though `flat_crossproduct` fails as well.

## Actual Behavior
CWLTool fails to provide the secondaryFile(s) to the scattered task, which then fails.

```
ERROR Exception on step 'dotTool'
ERROR Cannot make scatter job: Missing required secondary file 'happy1.txt.qt' from file object: {
"class": "File",
"location": "file:///Users/franklinmichael/Desktop/tmp/happy1.txt",
"size": 0,
"basename": "happy1.txt",
"nameroot": "happy1",
"nameext": ".txt",
"secondaryFiles": []
}
ERROR Workflow cannot make any more progress.
WARNING Final process status is permanentFail
```

## Workflow Code
```cwl
class: Workflow
cwlVersion: v1.0
id: sbmf
inputs:
inp1:
type:
items: File
type: array
inp2:
type:
items: string
type: array
outputs:
id:
outputSource: dotTool/out
type:
items: File
type: array
requirements:
InlineJavascriptRequirement: {}
ScatterFeatureRequirement: {}
StepInputExpressionRequirement: {}
steps:
dotTool:
in:
input1:
source: inp1
input2:
source: inp2
out:
- out
run: tools/TestStepTool.cwl
scatter:
- input1
- input2
scatterMethod: dotproduct
```

**TestStepTool.cwl**

Click to show tool

```cwl
baseCommand: echo
class: CommandLineTool
cwlVersion: v1.0
id: TestStepTool
inputs:
- id: input1
inputBinding:
position: 0
label: input1
secondaryFiles:
- .qt
type:
- File
- 'null'
- id: input2
inputBinding:
position: 1
label: input2
type:
- string
- 'null'
outputs:
- id: out
label: out
type: stdout
requirements:
DockerRequirement:
dockerPull: ubuntu:latest
```

**Job:**

Click to show File preparation

```bash
touch /path/to/happy1.txt
touch /path/to/happy1.txt.qt

touch /path/to/happy1.txt
touch /path/to/happy1.txt.qt
```

```yaml
inp1:
- class: File
path: /path/to/happy1.txt # happy1.txt.qt in same location
- class: File
path: /path/to/happy2.txt # happy2.txt.qt in same location
inp2:
- Janis
- branching
```

## Full Traceback

Click to show Traceback

```pytb
$ cwltool --debug --disable-color /Users/franklinmichael/janis/execution/sbmf/20191009_104956_e5cf44/workflow/sbmf.cwl /Users/franklinmichael/janis/execution/sbmf/20191009_104956_e5cf44/workflow/sbmf-inp.yml
INFO /anaconda3/bin/cwltool 1.0.20190906054215
INFO Resolved '/Users/franklinmichael/janis/execution/sbmf/20191009_104956_e5cf44/workflow/sbmf.cwl' to 'file:///Users/franklinmichael/janis/execution/sbmf/20191009_104956_e5cf44/workflow/sbmf.cwl'
DEBUG [workflow ] initialized from file:///Users/franklinmichael/janis/execution/sbmf/20191009_104956_e5cf44/workflow/sbmf.cwl#sbmf
INFO [workflow ] start
DEBUG [workflow ] {
"dotTool_runtime_cpu": 1,
"dotTool_runtime_disks": "local-disk 60 SSD",
"dotTool_runtime_memory": null,
"inp1": [
{
"class": "File",
"location": "file:///Users/franklinmichael/Desktop/tmp/happy1.txt",
"size": 0,
"basename": "happy1.txt",
"nameroot": "happy1",
"nameext": ".txt"
},
{
"class": "File",
"location": "file:///Users/franklinmichael/Desktop/tmp/happy2.txt",
"size": 0,
"basename": "happy2.txt",
"nameroot": "happy2",
"nameext": ".txt"
}
],
"inp2": [
"Janis",
"branching"
]
}
INFO [workflow ] starting step dotTool
INFO [step dotTool] start
ERROR Exception on step 'dotTool'
ERROR Cannot make scatter job: Missing required secondary file 'happy1.txt.qt' from file object: {
"class": "File",
"location": "file:///Users/franklinmichael/Desktop/tmp/happy1.txt",
"size": 0,
"basename": "happy1.txt",
"nameroot": "happy1",
"nameext": ".txt",
"secondaryFiles": []
}
DEBUG
Traceback (most recent call last):
File "/anaconda3/lib/python3.7/site-packages/cwltool/workflow.py", line 870, in parallel_steps
for j in step:
File "/anaconda3/lib/python3.7/site-packages/cwltool/workflow.py", line 208, in job
for j in self.step.job(joborder, output_callback, runtimeContext):
File "/anaconda3/lib/python3.7/site-packages/cwltool/workflow.py", line 807, in job
runtimeContext):
File "/anaconda3/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 430, in job
builder = self._init_job(job_order, runtimeContext)
File "/anaconda3/lib/python3.7/site-packages/cwltool/process.py", line 718, in _init_job
discover_secondaryFiles=getdefault(runtime_context.toplevel, False)))
File "/anaconda3/lib/python3.7/site-packages/cwltool/builder.py", line 276, in bind_input
bindings.extend(self.bind_input(f, datum[f["name"]], lead_pos=lead_pos, tail_pos=f["name"], discover_secondaryFiles=discover_secondaryFiles))
File "/anaconda3/lib/python3.7/site-packages/cwltool/builder.py", line 249, in bind_input
return self.bind_input(schema, datum, lead_pos=lead_pos, tail_pos=tail_pos, discover_secondaryFiles=discover_secondaryFiles)
File "/anaconda3/lib/python3.7/site-packages/cwltool/builder.py", line 342, in bind_input
sfname, json_dumps(datum, indent=4)))
cwltool.errors.WorkflowException: Missing required secondary file 'happy1.txt.qt' from file object: {
"class": "File",
"location": "file:///Users/franklinmichael/Desktop/tmp/happy1.txt",
"size": 0,
"basename": "happy1.txt",
"nameroot": "happy1",
"nameext": ".txt",
"secondaryFiles": []
}
ERROR Workflow cannot make any more progress.
WARNING Final process status is permanentFail
```

## Your Environment
* cwltool version: `/anaconda3/bin/cwltool 1.0.20190906054215`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.