Cancelling a backfill leaves its runs unschedulable forever
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.2
### What happened and how to reproduce it?
After a backfill is cancelled, clearing one of its DAG runs sends the run back to `queued`, where it
stays. In this case it had still not started four days later, with plenty of idle capacity, and the
scheduler had never even looked at it: `last_scheduling_decision` was still NULL and there was not a
single log line mentioning the run.
Cancelling sets both `is_paused=True` and `completed_at` on the backfill, and never clears `is_paused`.
The scheduler's promotion query only looks at the flag:
```python
# DagRun.get_queued_dag_runs_to_set_running
not_(coalesce(cast("ColumnElement[bool]", Backfill.is_paused), False)),
```
So the flag is stale but permanent, and every run still attached to that backfill is skipped on every
loop. `PUT /backfills/{id}/unpause` cannot help either, it returns `409 Backfill is already completed.`
**How to reproduce**
```python
from datetime import datetime
from airflow.sdk import dag, task
@dag(dag_id="cancelled_backfill_strands_run", schedule="@hourly",
start_date=datetime(2026, 1, 1), catchup=False)
def cancelled_backfill_strands_run():
@task
def work(): pass
work()
cancelled_backfill_strands_run()
```
1. Create a backfill over a past window and let at least one run finish.
2. `PUT /api/v2/backfills/{id}/cancel`.
3. Clear that finished run.
4. It returns to `queued` and never starts.
### What you think should happen instead?
Pausing a backfill should stop its runs while it is live. Once it has completed, `is_paused` is
meaningless and should not keep its runs out of the scheduler.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
**Suggested fix**
```python
or_(
Backfill.completed_at.isnot(None),
not_(coalesce(cast("ColumnElement[bool]", Backfill.is_paused), False)),
)
```
A live pause still behaves as before, only the completed-and-paused combination changes, and stranded runs
recover on the next loop with no manual intervention. PR #72054 already proposes this exact diff, though
its regression test is filed against #69658, which turned out to be the starvation issue rather than this
one.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Review DagRun.get_queued_dag_runs_to_set_running and PR #72054 first; inspect the regression test associated with #69658. Confirm that live paused backfills remain excluded while runs from completed backfills are considered on the next scheduler loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100