apache / apache/airflow

StatsD memory accumulation causes probe failures and restarts

Open
#50,645 4 comments 0 reactions 0 assignees View on GitHub
area:core area:helm-chart kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

Other Airflow 2 version (please specify below)

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

In Airflow deployments with many DAGs and task runs, we’ve observed that the statsd pod gradually consumes more memory and CPU over time. This seems to be caused by the continuous accumulation of metrics as Airflow processes more tasks and DAGs.

Currently, there doesn’t appear to be any cleanup or memory management mechanism in place for old or unused metrics. As a result, the resource usage of the pod keeps increasing until it becomes unresponsive, which causes the `/metrics` endpoint to fail and the pod to be restarted by Kubernetes.

![Image](https://github.com/user-attachments/assets/099a352f-dcb1-4558-bab7-0e3c64692f85)

### What you think should happen instead?

_No response_

### How to reproduce

1. Deploy Airflow via the official Helm chart with the statsd pod enabled.
2. Run a high volume of DAGs and tasks over time.
3. Allow default StatsD mappings to remain enabled.
4. Observe increasing memory usage in the statsd pod.
5. Eventually, the `/metrics `endpoint times out and the pod is restarted.

### Operating System

N/A

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

The report names no source files or tests; begin with the official Helm chart's StatsD pod, default StatsD mappings, and the /metrics endpoint. Reproduce the memory and CPU increase under sustained DAG and task load, then trace where metrics accumulate. Done means the endpoint remains responsive and the pod no longer grows until Kubernetes restarts it.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
infrastructure, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.