Removing DAG runs for a specific time range
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
Allow removing DAG runs from CLI or/and UI, additionally allowing to limit time range by specifying _before_ and/or period for logical dates. Eg. `airflow dags delete-runs --before 2025-01-01 00:00:00+01:00 --after 2024-01-01 00:00:00+01:00` or similar form in UI.
### Use case/motivation
I'm aware that in Airflow 3.0, individual DAG runs can be removed via the API. However, deleting a large number of DAG runs can be tedious and typically requires additional scripting to do efficiently. Having a built-in feature for this would be a helpful workaround for the following issues:
**Updating executor_config for historical runs**
In some cases, we need to rerun DAG runs for historical data intervals in scheduled DAGs. This may involve applying changes to the executor_config (e.g., updating Kubernetes worker resources or the Docker image). However, as described in [issue #26568](https://github.com/apache/airflow/issues/26568), the executor_config is saved in the database when the original DAG run is created. When a DAG run is cleared, this stored configuration is reused—so any updates to it are ignored.
A workaround is to delete the DAG run entirely. This forces the scheduler to re-parse the DAG definition and create a new DAG run with the updated executor_config.
**Reloading updated DAG structure**
If the DAG definition has changed (e.g., tasks have been added or removed), and you need to rerun historical DAG runs, there is a limitation in Airflow 3.0 when using versioned DAG bundles. In this case, clearing the DAG run won't reload the updated structure, since task instances are associated with the serialized DAG version that existed at the time of their creation.
If there are better ways to achieve these outcomes without deleting DAG runs, I’d appreciate any guidance on that.
### Related issues
[26568](https://github.com/apache/airflow/issues/26568)
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by tracing the existing DAG-run deletion entry points in the CLI and UI, then compare them with the Airflow 3.0 API behavior mentioned in the issue. Define how before and after logical-date bounds should work and verify that selected DAG runs are deleted without requiring additional scripting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data-engineering, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100