Statsd aggregation issue on some metrics with multiple schedulers running
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
Other Airflow 2 version
### What happened
When I am running multiple schedulers (>1) the statsd exporter does not correctly sum the running tasks from the different scheduler but it does just take one metric from one scheduler and display it as the value.
the metric name : `airflow_executor_running_tasks`
### What you think should happen instead
airflow statsd exporter should sum up all running tasks from all the existing and running schedulers to calculate `airflow_executor_running_tasks`
### How to reproduce
Run an airflow cluster with 2 schedulers and with statsd enabled.
Create enough tasks to balance them between the 2 schedulers. Using the UI you can check that between the displayed value and the `airflow_executor_running_tasks` metric exposed by statsd you will not have the same number?
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==3.4.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.2
apache-airflow-providers-docker==2.7.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==7.0.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.9.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Locate the StatsD emission and aggregation path for the airflow_executor_running_tasks metric, then reproduce the report with two schedulers and StatsD enabled. Done means the exposed metric equals the combined running-task count across both schedulers, while the existing UI value remains consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100