apache / apache/airflow

Airflow pools do not count tasks in scheduled state

Open
#69,187 2 comments 0 reactions 0 assignees View on GitHub
area:core kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.2.1

### What happened and how to reproduce it?

Pools that include deferred tasks do not count tasks in the scheduled state. For the first scheduled state over a task lifetime, I imagine this is not problematic, but whenever there are subsequent occurrences of the scheduled state in a deferred task's lifetime, any pool tracking that task is at risk of undercounting how many slots are occupied. This reduces the efficacy of pools as a semaphore protecting some external resource, which was the motivation of https://github.com/apache/airflow/pull/32709, the PR that enabled pools to track deferred tasks.

See the maintainer comment on a related issue: https://github.com/apache/airflow/issues/29416#issuecomment-1423685677.

Given that this is a matter of pool implementation, reproduction is trivial. Pools will undercount by definition: [source](https://github.com/apache/airflow/blob/bfd7689273d61dc9ed9b7fc3086e29e7abbd528c/airflow-core/src/airflow/models/pool.py#L245-L249).

### What you think should happen instead?

Per the [maintainer comment](https://github.com/apache/airflow/issues/29416#issuecomment-1423685677), and given that deferred tasks can transition into and out of the scheduled state arbitrarily many times, tasks in their second or greater scheduled state should count toward pool totals, provided the pool includes deferred tasks.

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow-core/src/airflow/models/pool.py at the linked lines and trace how deferred tasks and scheduled-state transitions contribute to pool totals. Reproduce the undercount described in the issue, then verify that a deferred task's second or later scheduled state is included when the pool tracks deferred tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.