StatsD memory accumulation causes probe failures and restarts
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
In Airflow deployments with many DAGs and task runs, we’ve observed that the statsd pod gradually consumes more memory and CPU over time. This seems to be caused by the continuous accumulation of metrics as Airflow processes more tasks and DAGs.
Currently, there doesn’t appear to be any cleanup or memory management mechanism in place for old or unused metrics. As a result, the resource usage of the pod keeps increasing until it becomes unresponsive, which causes the `/metrics` endpoint to fail and the pod to be restarted by Kubernetes.

### What you think should happen instead?
_No response_
### How to reproduce
1. Deploy Airflow via the official Helm chart with the statsd pod enabled.
2. Run a high volume of DAGs and tasks over time.
3. Allow default StatsD mappings to remain enabled.
4. Observe increasing memory usage in the statsd pod.
5. Eventually, the `/metrics `endpoint times out and the pod is restarted.
### Operating System
N/A
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
The report names no source files or tests; begin with the official Helm chart's StatsD pod, default StatsD mappings, and the /metrics endpoint. Reproduce the memory and CPU increase under sustained DAG and task load, then trace where metrics accumulate. Done means the endpoint remains responsive and the pod no longer grows until Kubernetes restarts it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- infrastructure, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100