apache / apache/airflow

Add support for automatic clearing of failed child DAG tasks in TriggerDagRunOperator

Open
#53,402 20 comments 0 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

I propose an enhancement to TriggerDagRunOperator that would allow automatic clearing of failed task instances in a triggered DAG run if that DAG run previously failed.

### Use case/motivation

When using TriggerDagRunOperator in synchronous mode (i.e., wait_for_completion=True), if the triggered child DAG fails (due to a failed task), the parent task also fails. However, when the operator is cleared in the UI (e.g., to retry), the associated failed child DAG run remains unchanged.

As a result:
- The retried TriggerDagRunOperator reuses the same failed DAG run,
- The child DAG run is still in failed state → the operator fails again,
- The user must manually clear failed tasks or the entire child DAG run.

This creates unnecessary friction when using DAG chaining.

### Proposed solution:

Introduce a new optional parameter:
```
TriggerDagRunOperator(
...,
auto_clear_failed_tasks=True
)
```
When set:
- Upon execution (or possibly in the clear() method logic),
- If a matching child DAG run exists and is in failed state,
- All failed task instances (and optionally downstreams) in that DAG run are automatically cleared.

This enables retrying the parent task to also retry the corresponding child DAG.

### Benefits
- Smoother reprocessing of chained DAGs
- Reduces manual intervention
- Fits naturally with Airflow's existing retry model

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with TriggerDagRunOperator and its existing clearing and retry behavior; the issue names no file or test. Resolve the matching child-run, failed-task/downstream scope, and execution-versus-clear-time semantics, then verify that retrying the parent handles a previously failed child run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.