apache / apache/airflow

Backfill premature completion (confirmed on Airflow 3.2.2)

Open
#68,721 3 comments 0 reactions 0 assignees View on GitHub
area:backfill area:core area:scheduler kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.2.2

### What happened and how to reproduce it?

Again more slop but hopefully useful enough.

`<🤖>`
---

When a backfill is created for a DAG with fast-completing tasks (sub-second per run),
the scheduler marks the backfill as complete before all queued runs have been executed.

The root cause is in `_mark_backfills_complete` (`scheduler_job_runner.py` ~line 1967),
which runs every 30 seconds and marks a backfill complete when no dag runs are in
`running` or `queued` state:

```python
~exists(
select(DagRun.id).where(
and_(DagRun.backfill_id == Backfill.id, DagRun.state.in_(unfinished_states))
)
)
```

When tasks complete faster than the scheduler's next scheduling loop can queue new
runs, there is a window where all current runs are `success` and the next batch has
not yet been dispatched. The completion check fires in this window and incorrectly
marks the backfill done, leaving remaining queued runs stranded.

**To reproduce:**

1. Create a DAG with a no-op task and a long date range:

```python
from airflow.sdk import dag, task
from datetime import datetime

@dag(
dag_id="test_backfill_bug",
schedule="@daily",
start_date=datetime(2020, 1, 1),
end_date=datetime(2022, 12, 31),
catchup=False,
)
def test_backfill_bug():
@task
def noop():
pass
noop()

test_backfill_bug()
```

2. Create a backfill:

```bash
airflow backfill create \
--dag-id test_backfill_bug \
--from-date 2020-01-01 \
--to-date 2022-12-31 \
--max-active-runs 10
```

3. Observe that the backfill completes having only processed a fraction of the 1096 runs:

```python
import sqlite3, os
conn = sqlite3.connect(os.path.expanduser('~/airflow/airflow.db'))
cur = conn.cursor()
cur.execute('SELECT id, completed_at FROM backfill WHERE dag_id="test_backfill_bug"')
b_id, completed_at = cur.fetchone()
cur.execute('SELECT state, COUNT(*) FROM dag_run WHERE backfill_id=? GROUP BY state', (b_id,))
print('completed_at:', completed_at)
for row in cur.fetchall(): print(row)
conn.close()
```

**Observed output:**

```
completed_at: 2026-06-18 03:42:47.759611
('success', 441)
('queued', 455) <- remaining runs never executed
```

### What you think should happen instead?

The backfill should only be marked complete when all dag runs associated with it have
reached a terminal state (`success` or `failed`), regardless of whether there is a
momentary window where none are `running` or `queued`.

A possible fix: check that the count of terminal dag runs equals the total
`BackfillDagRun` associations (excluding skipped entries) before marking complete.

### Operating System

macOS

### Deployment

Virtualenv installation

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

Note: PR #62561 (merged in 3.2.2) fixed a related but distinct issue where a backfill
was marked complete before *any* dag runs were created (zero-runs race). This issue
occurs after dag runs are created and begin executing — the completion window opens
between scheduling batches when tasks complete faster than new ones are dispatched.

Related issue: the SQLite `database is locked` error (see
[apache/airflow#68699](https://github.com/apache/airflow/issues/68699)) can cause fewer
dag runs to be created than expected, which makes this bug easier to trigger since
fewer runs complete faster.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading _mark_backfills_complete in scheduler_job_runner.py around line 1967, then run the supplied fast-task DAG and airflow backfill create reproduction. Verify the backfill does not complete while associated runs remain queued, and that completion occurs only after all non-skipped runs reach success or failed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.