common-workflow-language / common-workflow-language/cwltool
cwltool:ProcessGenerator output
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
According to the ProcessGenerator documentation, https://github.com/common-workflow-language/cwltool/blob/main/docs/processgen.rst
> The output of the generated script is used as the output for ProcessGenerator as a whole.
However, it appears that the final output is the generated process itself (i.e. the output of the embedded script), rather than the output of the generated process.
```
cwltool --enable-dev --enable-ext pytoolgen.txt zing.txt
INFO /Users/jakefennick/miniconda3/envs/base/bin/cwltool 3.1.20221201130942
INFO Resolved 'pytoolgen.txt' to 'file:///Users/jakefennick/pytoolgen.txt'
INFO [job cc371ad5-a371-4cf8-a514-28e56d3e15c0] /private/tmp/docker_tmpt87k7ul2$ python \
inp.py > /private/tmp/docker_tmpt87k7ul2/main.cwl
INFO [job cc371ad5-a371-4cf8-a514-28e56d3e15c0] completed success
INFO [job main.cwl] /private/tmp/docker_tmp0h47zegt$ echo \
blurf
blurf
INFO [job main.cwl] completed success
{
"runProcess": {
"location": "file:///Users/jakefennick/main.cwl",
"basename": "main.cwl",
"class": "File",
"checksum": "sha1$0cbf24b48a78fb6559954b959da950120aeb514f",
"size": 97,
"path": "/Users/jakefennick/main.cwl"
}
}
INFO Final process status is success
```
It should be noted that the attached pytoolgen.txt explicitly contains `outputs: {}`, which is apparently ignored.
Moreover, if any other top-level `outputs` are given (even constant literals) they are again ignored, along with the output of the generated process.
```
outputs:
myoutput:
type: string
outputBinding:
outputEval: $("myoutput")
```
```
cwltool --enable-dev --enable-ext pytoolgen_myoutput.txt zing.txt
INFO /Users/jakefennick/miniconda3/envs/base/bin/cwltool 3.1.20221201130942
INFO Resolved 'pytoolgen_myoutput.txt' to 'file:///Users/jakefennick/pytoolgen_myoutput.txt'
URI prefix 'cwltool' of 'cwltool:loop' not recognized, are you missing a $namespaces section?
INFO [job _:8190dc70-d058-42c3-a9cc-4044df633669] /private/tmp/docker_tmpx2398i1e$ python \
inp.py > /private/tmp/docker_tmpx2398i1e/main.cwl
INFO [job _:8190dc70-d058-42c3-a9cc-4044df633669] completed success
INFO [job main.cwl] /private/tmp/docker_tmpvxeig3jm$ echo \
blurf
blurf
INFO [job main.cwl] completed success
{
"runProcess": {
"location": "file:///Users/jakefennick/main.cwl",
"basename": "main.cwl",
"class": "File",
"checksum": "sha1$0cbf24b48a78fb6559954b959da950120aeb514f",
"size": 97,
"path": "/Users/jakefennick/main.cwl"
}
}
INFO Final process status is success
```
The plot thickens if we try to wrap a ProcessGenerator in a single step workflow:
```
cwltool --enable-dev --enable-ext pytoolgen_wrap.txt zing.txt
INFO /Users/jakefennick/miniconda3/envs/base/bin/cwltool 3.1.20221201130942
INFO Resolved 'pytoolgen_wrap.txt' to 'file:///Users/jakefennick/pytoolgen_wrap.txt'
URI prefix 'cwltool' of 'cwltool:loop' not recognized, are you missing a $namespaces section?
ERROR Tool definition failed validation:
pytoolgen_wrap.txt:8:9: Workflow step output 'runProcess' does not correspond to
pytoolgen.txt:8:1: tool output (expected '')
```
So it appears that the `outputs` are NOT being ignored w.r.t. a parent workflow. Moreover, if we explicitly shadow the (correct?) output, we get a different error:
```
cwltool --enable-dev --enable-ext pytoolgen_wrap_runProcess.txt zing.txt
INFO /Users/jakefennick/miniconda3/envs/base/bin/cwltool 3.1.20221201130942
INFO Resolved 'pytoolgen_wrap_runProcess.txt' to 'file:///Users/jakefennick/pytoolgen_wrap_runProcess.txt'
URI prefix 'cwltool' of 'cwltool:loop' not recognized, are you missing a $namespaces section?
INFO [workflow ] start
INFO [workflow ] starting step step1
INFO [step step1] start
INFO [job step1] /private/tmp/docker_tmpk0zpx3sq$ python \
inp.py > /private/tmp/docker_tmpk0zpx3sq/main.cwl
INFO [job step1] completed success
INFO [job step1_2] /private/tmp/docker_tmprbbqaggh$ echo \
blurf
blurf
INFO [job step1_2] completed success
ERROR [step step1] Output is missing expected field file:///Users/jakefennick/pytoolgen_wrap_runProcess.txt#step1/runProcess
WARNING [step step1] completed permanentFail
INFO [workflow ] completed permanentFail
{
"runProcess": null
}
WARNING Final process status is permanentFail
```
## Expected Behavior
pytoolget.txt should return the output from the generated process, not the generated process itself.
pytoolgen_myoutput.txt should also return "myoutput"
pytoolgen_wrap.txt and pytoolgen_wrap_runProcess.txt should return the same output as the underlying ProcessGenerator (whatever that might be).
## Actual Behavior
In both cases, the generated process itself is returned, not its output.
pytoolgen_wrap.txt and pytoolgen_wrap_runProcess.txt return the above errors.
## Workflow Code
[pytoolgen_runProcess.txt](https://github.com/common-workflow-language/cwltool/files/10925939/pytoolgen_runProcess.txt)
[pytoolgen_myoutput.txt](https://github.com/common-workflow-language/cwltool/files/10925940/pytoolgen_myoutput.txt)
[pytoolgen_wrap_runProcess.txt](https://github.com/common-workflow-language/cwltool/files/10925941/pytoolgen_wrap_runProcess.txt)
[pytoolgen.txt](https://github.com/common-workflow-language/cwltool/files/10925942/pytoolgen.txt)
[pytoolgen_wrap.txt](https://github.com/common-workflow-language/cwltool/files/10925943/pytoolgen_wrap.txt)
[zing.txt](https://github.com/common-workflow-language/cwltool/files/10925739/zing.txt)
## Full Traceback
See above
## Your Environment
* cwltool version: 3.1.20221201130942
Contributor guide
Assessment
This issue has not been assessed yet.