argoproj / argoproj/argo-workflows
Ability to specify outputs in DAG tasks
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 138
Description
# Summary
What change needs making?
Right now outputs can only be specified in base template, DAG & Step tasks can only pass arguments. Would be great if DAG tasks could specify the outputs, so the base template(container/script) can be re-used amongst tasks.
# Use Cases
Example:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-outputs
spec:
entrypoint: bake-me-a-cake
templates:
- name: py-runner
inputs:
parameters:
- name: source
script:
command:
- python3.6
image: alpine:latest
imagePullPolicy: Always
source: '{{inputs.parameters.source}}'
- name: bake-me-a-cake
dag:
tasks:
- name: mix
depends: ''
template: py-runner
arguments:
parameters:
- name: source
value: |-
with open("/tmp/ingredients", "w") as f:
f.write("sugar,flour,milk,chocolate")
outputs:
parameters:
- name: ingredients
valueFrom:
path: "/tmp/ingredients"
- name: bake
depends: 'mix'
template: py-runner
arguments:
parameters:
- name: source
value: |-
ingredients = "{{tasks.mix.outputs.parameters.ingredients}}"
serve = print(f"bake(ingredients)")
with open("/tmp/serve", "w") as f:
f.write("serve")
outputs:
parameters:
- name: serve
valueFrom:
path: "/tmp/serve"
- name: serve
depends: 'bake'
template: py-runner
arguments:
parameters:
- name: source
value: |-
serve = "{{tasks.bake.outputs.parameters.serve}}"
print(serve)
```
Basically this avoids having to duplicate script template over and over again just to serve different/multiple outputs. All that's different are the output paths in each DAG task. This makes it very easy to stitch complex workflows together without a ton of boilerplate.
---
**Message from the maintainers**:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
Contributor guide
Research direction
Start with the YAML use case and compare how outputs are currently declared in the base template versus the DAG tasks. Done means DAG tasks can declare distinct output paths while reusing one script or container template, and downstream tasks can consume those outputs as shown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100