argoproj / argoproj/argo-workflows

Reference outputs of nested steps

Open
#2,302 4 comments 9 reactions 0 assignees View on GitHub
area/templates/steps area/templating
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 15h
Merged PRs (30d)
138

Description

# Summary

Suppose we have the following workflow:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: reference-nested-step-outputs-
spec:
entrypoint: root
templates:
- name: root
steps:
- - name: step-1
template: nested
- - name: step-2
template: echo
arguments:
parameters:
- name: message
value: "{{steps.step-1.steps.step-1c.outputs.result}}"

- name: nested
steps:
- - name: step-1a
template: echo
arguments:
parameters:
- name: message
value: "step-1a"
- name: step-1b
template: echo
arguments:
parameters:
- name: message
value: "step-1b"
- - name: step-1c
template: echo
arguments:
parameters:
- name: message
value: "{{steps.step-1a.outputs.result}} + {{steps.step-1b.outputs.result}}"

- name: echo
inputs:
parameters:
- name: message
script:
image: alpine
command: ["ash"]
source: |
echo {{inputs.parameters.message}}
```

If we comment out `step-2` and execute the workflow, this will succeed. Specifically `step-1c` composes the results of `step-1a` and `step-1b` by referencing their outputs as follows:
```
{{steps.step-1a.outputs.result}} + {{steps.step-1b.outputs.result}}
```

However, if we uncomment `step-2`, our workflow will fail. Is it possible to enable referencing of nested step outputs in a workflow?

# Motivation

We have a workflow that creates EMR clusters on demand. We need to do a bunch of `aws` commands in parallel, followed by a single `aws create-cluster` command. Downstream steps need to know the `cluster-id` created in this step for further processing. It would be nice to encapsulate the "create EMR" steps within one template, and then be able to reference these outputs downstream in other steps/templates.

# Proposal

Perhaps nested steps could be referenced as follows:
```
{{steps.parent-step.steps.child-step.outputs.result}}
```

I think steps should have access to their own scope as well as their children's scope, but not their parents scope. Handling references to outputs of conditional steps would be tricky, it may have to be up to the workflow author to ensure conditional step outputs are not referenced in a scenario where the condition evaluates to false.

---

**Message from the maintainers**:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Contributor guide

Open the contributing guide

Research direction

Reproduce the nested workflow shown in the issue, first with step-2 commented out and then enabled, to confirm the output-reference failure. Investigate the workflow's nested-step output resolution and define done as allowing the proposed child-output reference to work for the demonstrated downstream step.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.