apache / apache/airflow

Make Open Telemetry the default instead of StatsD for monitoring

Open
#40,800 26 comments 4 reactions 1 assignee Assigned to @dirrao View on GitHub
telemetry
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

Currently, Apache Airflow uses StatsD for metrics collection and monitoring. To modernize our observability stack and align with industry standards, we should adopt OpenTelemetry as the primary metrics collection and monitoring tool for Airflow 3 while keeping Statsd as an option. This is made easier with [AIP-49 Open Telemetry Support](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow?src=contextnavpagetreemode).

### Backward Compatibility / Migration
Most of the enterprise Airflow deployments rely on StatsD for monitoring & alerting so we should make sure that there is a smooth migration path. Ideally, all the StatsD metrics mentioned in https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#metric-descriptions should have a like-by-like replacement.

**High-level Objectives**:
1. Use OpenTelemetry as the default metrics collection tool. This includes setting up configurations, dependencies, and code integration.
2. Documentation Update: Update the official documentation to reflect the change from StatsD to OpenTelemetry as the default, including setup, configuration, and usage instructions. Include migration guide for users transitioning from StatsD to OpenTelemetry if there are breaking changes. This includes the Helm chart.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.