apache / apache/airflow

Cancelling a backfill leaves its runs unschedulable forever

Open
#72,337 0 comments 0 reactions 0 assignees View on GitHub
kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.2.2

### What happened and how to reproduce it?

After a backfill is cancelled, clearing one of its DAG runs sends the run back to `queued`, where it
stays. In this case it had still not started four days later, with plenty of idle capacity, and the
scheduler had never even looked at it: `last_scheduling_decision` was still NULL and there was not a
single log line mentioning the run.

Cancelling sets both `is_paused=True` and `completed_at` on the backfill, and never clears `is_paused`.
The scheduler's promotion query only looks at the flag:

```python
# DagRun.get_queued_dag_runs_to_set_running
not_(coalesce(cast("ColumnElement[bool]", Backfill.is_paused), False)),
```

So the flag is stale but permanent, and every run still attached to that backfill is skipped on every
loop. `PUT /backfills/{id}/unpause` cannot help either, it returns `409 Backfill is already completed.`

**How to reproduce**

```python
from datetime import datetime
from airflow.sdk import dag, task

@dag(dag_id="cancelled_backfill_strands_run", schedule="@hourly",
start_date=datetime(2026, 1, 1), catchup=False)
def cancelled_backfill_strands_run():
@task
def work(): pass
work()

cancelled_backfill_strands_run()
```

1. Create a backfill over a past window and let at least one run finish.
2. `PUT /api/v2/backfills/{id}/cancel`.
3. Clear that finished run.
4. It returns to `queued` and never starts.

### What you think should happen instead?

Pausing a backfill should stop its runs while it is live. Once it has completed, `is_paused` is
meaningless and should not keep its runs out of the scheduler.

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

**Suggested fix**

```python
or_(
Backfill.completed_at.isnot(None),
not_(coalesce(cast("ColumnElement[bool]", Backfill.is_paused), False)),
)
```

A live pause still behaves as before, only the completed-and-paused combination changes, and stranded runs
recover on the next loop with no manual intervention. PR #72054 already proposes this exact diff, though
its regression test is filed against #69658, which turned out to be the starvation issue rather than this
one.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Review DagRun.get_queued_dag_runs_to_set_running and PR #72054 first; inspect the regression test associated with #69658. Confirm that live paused backfills remain excluded while runs from completed backfills are considered on the next scheduler loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.