apache / apache/airflow

Provide configuration option to configure path of OTEL metrics exporter

Open Beginner friendly
#42,492 3 comments 7 reactions 0 assignees View on GitHub
good first issue kind:feature telemetry
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Currently Airflow does not allow to configure path in [OTEL metrics exporter](https://github.com/apache/airflow/blob/2.10.2/airflow/metrics/otel_logger.py#L391C1-L403C56)
```python
def get_otel_logger(cls) -> SafeOtelLogger:
host = conf.get("metrics", "otel_host") # ex: "breeze-otel-collector"
port = conf.getint("metrics", "otel_port") # ex: 4318
prefix = conf.get("metrics", "otel_prefix") # ex: "airflow"
ssl_active = conf.getboolean("metrics", "otel_ssl_active")
# PeriodicExportingMetricReader will default to an interval of 60000 millis.
interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None) # ex: 30000
debug = conf.getboolean("metrics", "otel_debugging_on")

resource = Resource(attributes={SERVICE_NAME: "Airflow"})

protocol = "https" if ssl_active else "http"
endpoint = f"{protocol}://{host}:{port}/v1/metrics"
```

And some of the metrics collectors, e.g. VictoriaMetrics expect the metrics to be sent to different path
- `/opentelemetry/api/v1/push`
- `/insert/0/opentelemetry/api/v1/push`

It will be great to have the metrics path to be configurable, like the following

```python
def get_otel_logger(cls) -> SafeOtelLogger:
host = conf.get("metrics", "otel_host") # ex: "breeze-otel-collector"
port = conf.getint("metrics", "otel_port") # ex: 4318
path = conf.get("metrics", "otel_path", fallback="/v1/metrics") # ex: "/opentelemetry/api/v1/push"
prefix = conf.get("metrics", "otel_prefix") # ex: "airflow"
ssl_active = conf.getboolean("metrics", "otel_ssl_active")
# PeriodicExportingMetricReader will default to an interval of 60000 millis.
interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None) # ex: 30000
debug = conf.getboolean("metrics", "otel_debugging_on")

resource = Resource(attributes={SERVICE_NAME: "Airflow"})

protocol = "https" if ssl_active else "http"
endpoint = f"{protocol}://{host}:{port}{path}"
```

### Use case/motivation

Ability to send metrics directly into OTEL-compatible metrics collectors, e.g. VictoriaMetrics.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow/metrics/otel_logger.py at get_otel_logger and review the existing metrics configuration entries used there. Add a configurable OTEL path with /v1/metrics as the default, then verify that endpoints can target the example collector paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.