Ability to Re-run Only Skipped Tasks from Airflow UI
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
Currently, Apache Airflow provides convenient UI options to clear and re-run failed tasks for a DAG run. However, there is no equivalent option to selectively re-run only tasks in the `skipped` state.
This becomes operationally challenging for large dynamic DAGs and TaskGroups where hundreds of tasks may exist.
### Use Case
We have a dynamically generated DAG where:
* tasks are created from metadata/configuration stored externally
* each task represents ingestion/loading for a table
* DAG contains 500+ dynamically generated tasks inside TaskGroups
* tasks execute in parallel using pools and concurrency controls
In some operational scenarios, tasks may enter the `skipped` state due to:
* transient resource limitations
* scheduler timing conditions
* dag run timeout
* temporary orchestration issues
Currently Airflow allows:
* re-running failed tasks
* clearing entire DAG runs
* manually selecting tasks one-by-one
However, there is no simple mechanism to:
> re-run only skipped tasks
### Problem
For large DAGs, manually selecting skipped tasks is operationally difficult and error-prone.
Re-running the full DAG is also inefficient because:
* successful tasks do not need re-execution
* only skipped tasks require recovery
* compute and runtime costs increase unnecessarily
### Proposed Enhancement
Add a UI and/or CLI capability to:
* filter tasks by `skipped` state
* clear and re-run only skipped tasks
Possible examples:
* "Clear Skipped Tasks"
* "Re-run Skipped Tasks"
* additional checkbox alongside existing failed/upstream_failed options
### Benefits
* Improves operational recovery workflows
* Reduces unnecessary task re-execution
* Helpful for dynamic DAGs and TaskGroups
* Improves usability for large-scale orchestration pipelines
### Environment
* Apache Airflow 2.7.2
* AWS MWAA
* Dynamic DAG generation using TaskGroups
* Parallel task execution using pools
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.