apache / apache/airflow

Make backfill creation robust against scheduler race condition

Open
#64,534 1 comment 0 reactions 0 assignees View on GitHub
area:backfill area:core area:scheduler
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

The `_create_backfill()` method in `airflow-core/src/airflow/models/backfill.py` creates a backfill in two steps:

1. It commits the `Backfill` row to the database (`session.commit()`)
2. Then it creates all the `DagRun` and `BackfillDagRun` rows linked to that backfill

The scheduler runs `_mark_backfills_complete()` every ~30 seconds, looking for backfills where all associated DagRuns have finished. If the scheduler runs between Step 1 and Step 2, it sees a backfill with zero DagRuns, concludes "all zero runs are done", and marks the backfill as complete. When Step 2 then creates the DagRuns, they're orphaned.

PR #62561 fixed this with a pragmatic guard: require at least one `BackfillDagRun` row before marking complete, with a 2-minute timeout to clean up orphaned backfills that failed during initialization. This works, but relies on a time-based heuristic rather than addressing the root cause.

During [PR #62561 review](https://github.com/apache/airflow/pull/62561#pullrequestreview-4030269060), Daniel Standish suggested two more robust approaches as follow-ups:

**Approach 1: Atomic transaction** — Wrap the entire backfill creation in a single transaction so `Backfill`, `DagRun`, and `BackfillDagRun` rows all appear atomically. One way: change `session.commit()` to `session.flush()` in `_create_backfill()` ([backfill.py L605](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/backfill.py#L605)). `flush()` assigns the ID without committing; `create_session()` commits everything at the end. Tradeoff: the [`AlreadyRunningBackfill` check](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/backfill.py#L585-L589) relies on the early commit to block concurrent duplicates, so this would need database-level locking. (Also suggested by [Kaxil](https://github.com/apache/airflow/pull/62561#discussion_r2880363898))

**Approach 2: Add a state field or datetime** — Add an explicit signal to distinguish "initializing" from "running" backfills. Either a nullable `started_at`/`initialized_at` datetime (set after DagRuns are created), or a state enum (`QUEUED`/`RUNNING`/`COMPLETED`/`FAILED`). Replaces the 2-minute heuristic with an explicit check. Requires a DB migration.

These approaches are not mutually exclusive. Either one alone improves on the current heuristic.

**Key code locations:**
- [`airflow-core/src/airflow/models/backfill.py`](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/backfill.py) — `Backfill` model and `_create_backfill()`
- [`airflow-core/src/airflow/jobs/scheduler_job_runner.py`](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py) — `_mark_backfills_complete()`

### Use case/motivation

The current fix in PR #62561 uses a 2-minute time-based heuristic to distinguish "still initializing" from "orphaned" backfills. This works but is fragile — it assumes backfill creation always completes within 2 minutes, and doesn't provide a clean way to detect or recover from failed initializations. A more robust solution would either eliminate the race window entirely (atomic transaction) or make the initialization state explicit (state field).

### Related issues

- #61375 — Original bug report (backfill marked complete before DagRuns are created)
- PR #62561 — Current fix with the EXISTS guard + time-based cutoff
- [Daniel's review comment](https://github.com/apache/airflow/pull/62561#pullrequestreview-4030269060) suggesting these follow-up approaches
- [Kaxil's review comment](https://github.com/apache/airflow/pull/62561#discussion_r2880363898) suggesting `flush()` instead of `commit()`

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.