Airflow pools do not count tasks in scheduled state
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.1
### What happened and how to reproduce it?
Pools that include deferred tasks do not count tasks in the scheduled state. For the first scheduled state over a task lifetime, I imagine this is not problematic, but whenever there are subsequent occurrences of the scheduled state in a deferred task's lifetime, any pool tracking that task is at risk of undercounting how many slots are occupied. This reduces the efficacy of pools as a semaphore protecting some external resource, which was the motivation of https://github.com/apache/airflow/pull/32709, the PR that enabled pools to track deferred tasks.
See the maintainer comment on a related issue: https://github.com/apache/airflow/issues/29416#issuecomment-1423685677.
Given that this is a matter of pool implementation, reproduction is trivial. Pools will undercount by definition: [source](https://github.com/apache/airflow/blob/bfd7689273d61dc9ed9b7fc3086e29e7abbd528c/airflow-core/src/airflow/models/pool.py#L245-L249).
### What you think should happen instead?
Per the [maintainer comment](https://github.com/apache/airflow/issues/29416#issuecomment-1423685677), and given that deferred tasks can transition into and out of the scheduled state arbitrarily many times, tasks in their second or greater scheduled state should count toward pool totals, provided the pool includes deferred tasks.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in airflow-core/src/airflow/models/pool.py at the linked lines and trace how deferred tasks and scheduled-state transitions contribute to pool totals. Reproduce the undercount described in the issue, then verify that a deferred task's second or later scheduled state is included when the pool tracks deferred tasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100