argoproj / argoproj/argo-workflows

Ability to specify outputs in DAG tasks

Open
#4,479 2 comments 2 reactions 0 assignees View on GitHub
area/templates/dag
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 15h
Merged PRs (30d)
138

Description

# Summary

What change needs making?

Right now outputs can only be specified in base template, DAG & Step tasks can only pass arguments. Would be great if DAG tasks could specify the outputs, so the base template(container/script) can be re-used amongst tasks.

# Use Cases

Example:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-outputs
spec:
entrypoint: bake-me-a-cake
templates:
- name: py-runner
inputs:
parameters:
- name: source
script:
command:
- python3.6
image: alpine:latest
imagePullPolicy: Always
source: '{{inputs.parameters.source}}'
- name: bake-me-a-cake
dag:
tasks:
- name: mix
depends: ''
template: py-runner
arguments:
parameters:
- name: source
value: |-
with open("/tmp/ingredients", "w") as f:
f.write("sugar,flour,milk,chocolate")
outputs:
parameters:
- name: ingredients
valueFrom:
path: "/tmp/ingredients"

- name: bake
depends: 'mix'
template: py-runner
arguments:
parameters:
- name: source
value: |-
ingredients = "{{tasks.mix.outputs.parameters.ingredients}}"
serve = print(f"bake(ingredients)")
with open("/tmp/serve", "w") as f:
f.write("serve")
outputs:
parameters:
- name: serve
valueFrom:
path: "/tmp/serve"

- name: serve
depends: 'bake'
template: py-runner
arguments:
parameters:
- name: source
value: |-
serve = "{{tasks.bake.outputs.parameters.serve}}"
print(serve)
```

Basically this avoids having to duplicate script template over and over again just to serve different/multiple outputs. All that's different are the output paths in each DAG task. This makes it very easy to stitch complex workflows together without a ton of boilerplate.

---

**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Contributor guide

Open the contributing guide

Research direction

Start with the YAML use case and compare how outputs are currently declared in the base template versus the DAG tasks. Done means DAG tasks can declare distinct output paths while reusing one script or container template, and downstream tasks can consume those outputs as shown.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.