apache / apache/airflow

Select version when re-running a dag/task

Open
#50,107 5 comments 0 reactions 0 assignees View on GitHub
area:core kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

3.0.0

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

When testing new dags, especially long-running dags, it's really frustrating to have to re-run a long dag just to be able to re-run the last task (e.g. there was a bug in the last task). Dag versioning is great, but it undermines the concept of task atomicity because in order to re-run a single task, I potentially need to re-run an entire otherwise perfectly functioning dag.

### What you think should happen instead?

Users should be able to select a dag version when clearing a task if that task existed in previous versions.

### How to reproduce

Create a dag run with a bug in the last task. Fix the bug, then clear the last task instance.

### Operating System

n/a

### Versions of Apache Airflow Providers

_No response_

### Deployment

Astronomer

### Deployment details

_No response_

### Anything else?

This is arguably a feature request, but I'm considering it a bug since this functionality exists in Airflow 2.x

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the task-clearing flow used when a task instance is cleared, then inspect how DAG versions are represented and selected during reruns. Done means a user can choose an earlier DAG version when clearing a task that existed in that version, without rerunning the entire DAG; add or update coverage for the described reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.