apache / apache/airflow

Scheduler stalls: 3.2 exceeds_max_non_backfill write contends with DAG-processor SELECT … FOR UPDATE on the dag table

Open
#70,614 1 comment 0 reactions 0 assignees View on GitHub
affected_version:3.2 area:backfill area:DAG-processing area:scheduler kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

3.2.0+ (observed on 3.2.2)

### What happened

The scheduler intermittently stalls: `scheduler.critical_section_duration` climbs while scheduler CPU drops and task scheduling starves. Inspecting the metadata DB during a stall shows scheduler statements of the form `UPDATE dag SET exceeds_max_non_backfill = … WHERE dag.dag_id = …` blocked with `wait_event_type = Lock` / `wait_event = transactionid`, waiting on the DAG processor's parse transaction.

### Root cause

Before 3.2, the scheduler evaluated `max_active_runs` in memory and did not write the `dag` row during dagrun scheduling. **#60006 ("Separate 'next dag run' from 'max active runs'", merge commit `32bc0119b408c9a37841edd021b646324a02745e`, shipped in 3.2.0)** added the `dag.exceeds_max_non_backfill` column (migration `0097_3_2_0_add_exceeds_max_runs_flag_to_dag_model`) and made the scheduler **write it to the `dag` row** during dagrun scheduling:

- Write: `airflow-core/src/airflow/jobs/scheduler_job_runner.py::_set_exceeds_max_active_runs` (called from `_create_dag_runs` and `_start_queued_dagruns`).

That write now competes with a pre-existing lock held by the DAG processor for the entire per-file parse transaction:

- Lock: `airflow-core/src/airflow/dag_processing/collection.py::DagModelOperation.find_orm_dags` → `with_row_locks(select(DagModel) …)` i.e. `SELECT … FOR UPDATE` on every `dag` row for a parsed file, held through `update_dags` + serialization + permission sync until the transaction commits.

So the scheduler's `UPDATE dag SET exceeds_max_non_backfill = …` blocks on the DAG processor's `FOR UPDATE`. This is amplified when a single parsed file defines many DAGs (e.g. dynamically generated DAGs), because that parse transaction can run for minutes — holding the `dag`-row locks the whole time and stalling the scheduler loop.

### How to reproduce it (conceptually)

1. Use the FAB auth manager and generate many DAGs from a single DAG file.
2. While the DAG processor is parsing that file (holding `SELECT … FOR UPDATE` on its `dag` rows), have the scheduler attempt to update `exceeds_max_non_backfill` for one of those DAGs.
3. Observe the scheduler statement blocked on `Lock`/`transactionid`, and `critical_section_duration` rising while scheduler CPU falls.

### Suggested direction

Reduce the parse transaction's `dag`-row lock hold and/or the scheduler's contention on it. As a partial step, per-DAG permission sync (another writer in the same parse transaction) can be made to honor `[fab] update_fab_perms` — proposed in #70589. A more complete fix would shorten/scope the `find_orm_dags` lock or the scheduler's `dag`-row write.

### Anything else

Related: #23232 (pluggable appless security manager).

Contributor guide

Open the contributing guide

Research direction

Start with airflow-core/src/airflow/jobs/scheduler_job_runner.py::_set_exceeds_max_active_runs and airflow-core/src/airflow/dag_processing/collection.py::DagModelOperation.find_orm_dags. Reproduce with the FAB auth manager and many DAGs generated from one file, observing row-lock waits and critical-section duration. Done means reducing the parse/scheduler contention so the scheduler no longer stalls during DAG parsing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.