apache / apache/airflow

Allow task to depend on specific tasks from previous dag_run

Open
#60,328 6 comments 0 reactions 0 assignees View on GitHub
area:core good first issue kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

depends_on_past=True currently allows a task to depend only on the same task in the previous dag_run.

There is no native way for a task to depend on the successful completion of multiple specific tasks from the previous dag_run (within the same DAG). Achieving this today requires custom sensors that directly query Airflow metadata tables.

### Use case/motivation

In stateful or partially retryable pipelines, it is common to require that a group of tasks in the previous dag_run has completed successfully before continuing execution in the next run.

Example:

`A → B → C → D → E`

If:
dag_run_1 fails at task D
Then for dag_run_2: Task C should not start until C, D, and E from dag_run_1 have all succeeded
Today:
depends_on_past is insufficient (checks only the same task)
trigger_rule applies only within the same dag_run
Users must implement custom PythonSensors that query DagRun / TaskInstance
This adds boilerplate, DB coupling, and operational complexity for a fairly common dependency pattern.

What would you like to happen?
Provide a declarative, first-class way for a task to depend on specific tasks from the previous dag_run.

Illustrative example (API shape open for discussion):
```
C = PythonOperator(
task_id="C",
depends_on_past=True,
depends_on_previous_task_ids=["C", "D", "E"],
)

```

> Semantics:

In dag_run(N), task C is scheduled only if the listed tasks in dag_run(N-1) all finished with success
First dag_run is not blocked
Backward compatible and fully opt-in
This would remove the need for custom sensors while keeping cross-run dependencies explicit and readable.

### Related issues

No known existing issue covering this specific capability.

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing how depends_on_past is evaluated for PythonOperator tasks, then review the DagRun and TaskInstance interactions mentioned in the issue. Define and validate an opt-in API for listing previous-run task IDs, including first-run behavior, success requirements, and backward compatibility; done means the dependency works without custom PythonSensors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.