apache / apache/airflow

Ability to Re-run Only Skipped Tasks from Airflow UI

Open
#67,464 1 comment 0 reactions 0 assignees View on GitHub
area:UI kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

Currently, Apache Airflow provides convenient UI options to clear and re-run failed tasks for a DAG run. However, there is no equivalent option to selectively re-run only tasks in the `skipped` state.

This becomes operationally challenging for large dynamic DAGs and TaskGroups where hundreds of tasks may exist.

### Use Case

We have a dynamically generated DAG where:

* tasks are created from metadata/configuration stored externally
* each task represents ingestion/loading for a table
* DAG contains 500+ dynamically generated tasks inside TaskGroups
* tasks execute in parallel using pools and concurrency controls

In some operational scenarios, tasks may enter the `skipped` state due to:

* transient resource limitations
* scheduler timing conditions
* dag run timeout
* temporary orchestration issues

Currently Airflow allows:

* re-running failed tasks
* clearing entire DAG runs
* manually selecting tasks one-by-one

However, there is no simple mechanism to:

> re-run only skipped tasks

### Problem

For large DAGs, manually selecting skipped tasks is operationally difficult and error-prone.

Re-running the full DAG is also inefficient because:

* successful tasks do not need re-execution
* only skipped tasks require recovery
* compute and runtime costs increase unnecessarily

### Proposed Enhancement

Add a UI and/or CLI capability to:

* filter tasks by `skipped` state
* clear and re-run only skipped tasks

Possible examples:

* "Clear Skipped Tasks"
* "Re-run Skipped Tasks"
* additional checkbox alongside existing failed/upstream_failed options

### Benefits

* Improves operational recovery workflows
* Reduces unnecessary task re-execution
* Helpful for dynamic DAGs and TaskGroups
* Improves usability for large-scale orchestration pipelines

### Environment

* Apache Airflow 2.7.2
* AWS MWAA
* Dynamic DAG generation using TaskGroups
* Parallel task execution using pools

Image

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.