argoproj / argoproj/argo-workflows
Reference outputs of nested steps
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 138
Description
# Summary
Suppose we have the following workflow:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: reference-nested-step-outputs-
spec:
entrypoint: root
templates:
- name: root
steps:
- - name: step-1
template: nested
- - name: step-2
template: echo
arguments:
parameters:
- name: message
value: "{{steps.step-1.steps.step-1c.outputs.result}}"
- name: nested
steps:
- - name: step-1a
template: echo
arguments:
parameters:
- name: message
value: "step-1a"
- name: step-1b
template: echo
arguments:
parameters:
- name: message
value: "step-1b"
- - name: step-1c
template: echo
arguments:
parameters:
- name: message
value: "{{steps.step-1a.outputs.result}} + {{steps.step-1b.outputs.result}}"
- name: echo
inputs:
parameters:
- name: message
script:
image: alpine
command: ["ash"]
source: |
echo {{inputs.parameters.message}}
```
If we comment out `step-2` and execute the workflow, this will succeed. Specifically `step-1c` composes the results of `step-1a` and `step-1b` by referencing their outputs as follows:
```
{{steps.step-1a.outputs.result}} + {{steps.step-1b.outputs.result}}
```
However, if we uncomment `step-2`, our workflow will fail. Is it possible to enable referencing of nested step outputs in a workflow?
# Motivation
We have a workflow that creates EMR clusters on demand. We need to do a bunch of `aws` commands in parallel, followed by a single `aws create-cluster` command. Downstream steps need to know the `cluster-id` created in this step for further processing. It would be nice to encapsulate the "create EMR" steps within one template, and then be able to reference these outputs downstream in other steps/templates.
# Proposal
Perhaps nested steps could be referenced as follows:
```
{{steps.parent-step.steps.child-step.outputs.result}}
```
I think steps should have access to their own scope as well as their children's scope, but not their parents scope. Handling references to outputs of conditional steps would be tricky, it may have to be up to the workflow author to ensure conditional step outputs are not referenced in a scenario where the condition evaluates to false.
---
**Message from the maintainers**:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
Contributor guide
Research direction
Reproduce the nested workflow shown in the issue, first with step-2 commented out and then enabled, to confirm the output-reference failure. Investigate the workflow's nested-step output resolution and define done as allowing the proposed child-output reference to work for the demonstrated downstream step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100