argoproj / argoproj/argo-workflows
Infer DAG dependencies from artifact/parameter references by default
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
# Summary
Many DAGs have a simple dependence structure: If step B uses step A's output (result, parameters or artifacts), then step A must be in step B's dependencies. Rather than duplicating this information in the `dependencies` (or `depends`) field, we could save lines of YAML to, by default, infer it from references like `{{tasks.earlier-step.outputs.artifacts.transformed-data}}` within a particular task.
# Use Cases
This would be useful for all sorts of data processing / ML pipelines, where artifact passing is the main reason for dependencies between steps. It would save users the effort of explicitly writing out these inferred dependencies or catching them via the linter.
---
**Message from the maintainers**:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
Contributor guide
Research direction
Start by tracing DAG dependency resolution and the linter behavior around the dependencies/depends field and task output references such as {{tasks.earlier-step.outputs.artifacts.transformed-data}}. Done means dependencies are inferred by default when a task references another task's results, parameters, or artifacts, without requiring duplicate declarations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100