Tasks remain in queued state for ~950s despite available worker capacity and no resource bottlenecks
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.9.0
### What happened?
In our production environment using `CeleryExecutor` and Redis as the broker, we’re experiencing intermittent but repeatable delays where tasks remain in the `queued` state for approximately 950 seconds before they begin execution.
This behavior is unexpected given that:
- There is plenty of free capacity in the pools used by the tasks.
- Four Celery workers are running, each with a concurrency of 32, for a total of 128 available slots.
- The Redis broker is not under high CPU/memory pressure.
- The are 4 schedulers (multi-scheduler enabled) and their heartbeat is healthy and operational.
- We verified the tasks are coming from different DAGs and using different pools.
- There is no CPU saturation on any component (scheduler, Redis, workers).
- The logs from the scheduler and workers show no errors or timeouts.
- The configuration key `AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT` is set to 3600.
Despite this, some tasks get queued and then do not start executing until ~950 seconds later, even when there is no obvious bottleneck or backlog.

### What you think should happen instead?
We expected tasks to transition from `queued` to `running` within a few seconds, as there are ample resources and capacity available.
### How to reproduce
We don't have a minimal reproducible example, but the issue appears randomly and independently of DAG size or the number of enqueued tasks. For example:
1. A DAG triggers one or more tasks.
2. Some tasks transition from `queued` to `running` quickly as expected.
3. Others remain stuck in `queued` for approximately 950 seconds, even when:
- Pools have available slots.
- Workers are idle or underutilized.
- Redis is healthy and not under pressure.
- There is no backlog of scheduled or queued tasks.
This behavior has been observed:
- Across multiple DAGs.
- Across different pools.
- Even when very few tasks are running in the system.
### Operating System
Ubuntu 22.04
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.6.1
apache-airflow-providers-common-io==1.3.0
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-fab==1.0.2
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-redis==3.6.0
apache-airflow-providers-smtp==1.6.1
apache-airflow-providers-sqlite==3.7.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
No source file, test, or entry point is identified. Start by reproducing the delay with Airflow 2.9.0, CeleryExecutor, Redis, and the listed deployment details, then inspect scheduler, worker, and broker behavior around the queued-to-running transition. Done means identifying the cause of the approximately 950-second delay and verifying a fix with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100