Scheduler stalls: 3.2 exceeds_max_non_backfill write contends with DAG-processor SELECT … FOR UPDATE on the dag table
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
3.2.0+ (observed on 3.2.2)
### What happened
The scheduler intermittently stalls: `scheduler.critical_section_duration` climbs while scheduler CPU drops and task scheduling starves. Inspecting the metadata DB during a stall shows scheduler statements of the form `UPDATE dag SET exceeds_max_non_backfill = … WHERE dag.dag_id = …` blocked with `wait_event_type = Lock` / `wait_event = transactionid`, waiting on the DAG processor's parse transaction.
### Root cause
Before 3.2, the scheduler evaluated `max_active_runs` in memory and did not write the `dag` row during dagrun scheduling. **#60006 ("Separate 'next dag run' from 'max active runs'", merge commit `32bc0119b408c9a37841edd021b646324a02745e`, shipped in 3.2.0)** added the `dag.exceeds_max_non_backfill` column (migration `0097_3_2_0_add_exceeds_max_runs_flag_to_dag_model`) and made the scheduler **write it to the `dag` row** during dagrun scheduling:
- Write: `airflow-core/src/airflow/jobs/scheduler_job_runner.py::_set_exceeds_max_active_runs` (called from `_create_dag_runs` and `_start_queued_dagruns`).
That write now competes with a pre-existing lock held by the DAG processor for the entire per-file parse transaction:
- Lock: `airflow-core/src/airflow/dag_processing/collection.py::DagModelOperation.find_orm_dags` → `with_row_locks(select(DagModel) …)` i.e. `SELECT … FOR UPDATE` on every `dag` row for a parsed file, held through `update_dags` + serialization + permission sync until the transaction commits.
So the scheduler's `UPDATE dag SET exceeds_max_non_backfill = …` blocks on the DAG processor's `FOR UPDATE`. This is amplified when a single parsed file defines many DAGs (e.g. dynamically generated DAGs), because that parse transaction can run for minutes — holding the `dag`-row locks the whole time and stalling the scheduler loop.
### How to reproduce it (conceptually)
1. Use the FAB auth manager and generate many DAGs from a single DAG file.
2. While the DAG processor is parsing that file (holding `SELECT … FOR UPDATE` on its `dag` rows), have the scheduler attempt to update `exceeds_max_non_backfill` for one of those DAGs.
3. Observe the scheduler statement blocked on `Lock`/`transactionid`, and `critical_section_duration` rising while scheduler CPU falls.
### Suggested direction
Reduce the parse transaction's `dag`-row lock hold and/or the scheduler's contention on it. As a partial step, per-DAG permission sync (another writer in the same parse transaction) can be made to honor `[fab] update_fab_perms` — proposed in #70589. A more complete fix would shorten/scope the `find_orm_dags` lock or the scheduler's `dag`-row write.
### Anything else
Related: #23232 (pluggable appless security manager).
Contributor guide
Research direction
Start with airflow-core/src/airflow/jobs/scheduler_job_runner.py::_set_exceeds_max_active_runs and airflow-core/src/airflow/dag_processing/collection.py::DagModelOperation.find_orm_dags. Reproduce with the FAB auth manager and many DAGs generated from one file, observing row-lock waits and critical-section duration. Done means reducing the parse/scheduler contention so the scheduler no longer stalls during DAG parsing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100