Allow task to depend on specific tasks from previous dag_run
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
depends_on_past=True currently allows a task to depend only on the same task in the previous dag_run.
There is no native way for a task to depend on the successful completion of multiple specific tasks from the previous dag_run (within the same DAG). Achieving this today requires custom sensors that directly query Airflow metadata tables.
### Use case/motivation
In stateful or partially retryable pipelines, it is common to require that a group of tasks in the previous dag_run has completed successfully before continuing execution in the next run.
Example:
`A → B → C → D → E`
If:
dag_run_1 fails at task D
Then for dag_run_2: Task C should not start until C, D, and E from dag_run_1 have all succeeded
Today:
depends_on_past is insufficient (checks only the same task)
trigger_rule applies only within the same dag_run
Users must implement custom PythonSensors that query DagRun / TaskInstance
This adds boilerplate, DB coupling, and operational complexity for a fairly common dependency pattern.
What would you like to happen?
Provide a declarative, first-class way for a task to depend on specific tasks from the previous dag_run.
Illustrative example (API shape open for discussion):
```
C = PythonOperator(
task_id="C",
depends_on_past=True,
depends_on_previous_task_ids=["C", "D", "E"],
)
```
> Semantics:
In dag_run(N), task C is scheduled only if the listed tasks in dag_run(N-1) all finished with success
First dag_run is not blocked
Backward compatible and fully opt-in
This would remove the need for custom sensors while keeping cross-run dependencies explicit and readable.
### Related issues
No known existing issue covering this specific capability.
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by tracing how depends_on_past is evaluated for PythonOperator tasks, then review the DagRun and TaskInstance interactions mentioned in the issue. Define and validate an opt-in API for listing previous-run task IDs, including first-run behavior, success requirements, and backward compatibility; done means the dependency works without custom PythonSensors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100