argoproj / argoproj/argo-workflows

Infer DAG dependencies from artifact/parameter references by default

Open
#5,255 0 comments 3 reactions 0 assignees View on GitHub
area/templates/dag area/templating
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

# Summary

Many DAGs have a simple dependence structure: If step B uses step A's output (result, parameters or artifacts), then step A must be in step B's dependencies. Rather than duplicating this information in the `dependencies` (or `depends`) field, we could save lines of YAML to, by default, infer it from references like `{{tasks.earlier-step.outputs.artifacts.transformed-data}}` within a particular task.

# Use Cases

This would be useful for all sorts of data processing / ML pipelines, where artifact passing is the main reason for dependencies between steps. It would save users the effort of explicitly writing out these inferred dependencies or catching them via the linter.

---

**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Contributor guide

Open the contributing guide

Research direction

Start by tracing DAG dependency resolution and the linter behavior around the dependencies/depends field and task output references such as {{tasks.earlier-step.outputs.artifacts.transformed-data}}. Done means dependencies are inferred by default when a task references another task's results, parameters, or artifacts, without requiring duplicate declarations.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.