Enhance max_active_runs log to include blocking Run IDs and Task IDs
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
Currently, when a DAG reaches its max_active_runs limit, the scheduler logs the following generic message:
`DAG is at (or above) max_active_runs ( of ), not creating any more runs`
While this indicates that the limit has been reached, it does not provide any context about what is occupying those active slots. To figure out why the DAG is stuck, users currently have to manually navigate through the Airflow UI or web server logs.
This contribution enhances the log message in scheduler_job_runner.py. When the max_active_runs limit is hit, the scheduler will now perform targeted queries to fetch the specific run_ids and task_ids (along with their current states) that are holding the active slots.
`DAG test_dag is at (or above) max_active_runs (1 of 1), not creating any more runs. Active Runs: scheduled__2023-10-18T09:55:00+00:00 | Active Tasks: [sleep_task in scheduled__2023-10-18T09:55:00+00:00 (running)]`
Implementation Note:
To avoid unnecessary database load, the queries to fetch the active runs and tasks are placed strictly inside the if total_active_runs >= dag.max_active_runs: condition. This ensures there is zero performance overhead during normal, healthy scheduling loops.
### Use case/motivation
By printing the exact Run IDs and Task IDs directly into the scheduler logs, platform administrators and data engineers can instantly identify the bottleneck without leaving their logging environment. This drastically reduces Time To Resolution for stuck DAGs.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in scheduler_job_runner.py at the max_active_runs condition and trace how active runs and tasks are queried. Verify that the scheduler log includes the blocking run IDs and task IDs with their states when the limit is reached, while no extra queries occur during normal scheduling loops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100