Provide configuration option to configure path of OTEL metrics exporter
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Currently Airflow does not allow to configure path in [OTEL metrics exporter](https://github.com/apache/airflow/blob/2.10.2/airflow/metrics/otel_logger.py#L391C1-L403C56)
```python
def get_otel_logger(cls) -> SafeOtelLogger:
host = conf.get("metrics", "otel_host") # ex: "breeze-otel-collector"
port = conf.getint("metrics", "otel_port") # ex: 4318
prefix = conf.get("metrics", "otel_prefix") # ex: "airflow"
ssl_active = conf.getboolean("metrics", "otel_ssl_active")
# PeriodicExportingMetricReader will default to an interval of 60000 millis.
interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None) # ex: 30000
debug = conf.getboolean("metrics", "otel_debugging_on")
resource = Resource(attributes={SERVICE_NAME: "Airflow"})
protocol = "https" if ssl_active else "http"
endpoint = f"{protocol}://{host}:{port}/v1/metrics"
```
And some of the metrics collectors, e.g. VictoriaMetrics expect the metrics to be sent to different path
- `/opentelemetry/api/v1/push`
- `/insert/0/opentelemetry/api/v1/push`
It will be great to have the metrics path to be configurable, like the following
```python
def get_otel_logger(cls) -> SafeOtelLogger:
host = conf.get("metrics", "otel_host") # ex: "breeze-otel-collector"
port = conf.getint("metrics", "otel_port") # ex: 4318
path = conf.get("metrics", "otel_path", fallback="/v1/metrics") # ex: "/opentelemetry/api/v1/push"
prefix = conf.get("metrics", "otel_prefix") # ex: "airflow"
ssl_active = conf.getboolean("metrics", "otel_ssl_active")
# PeriodicExportingMetricReader will default to an interval of 60000 millis.
interval = conf.getint("metrics", "otel_interval_milliseconds", fallback=None) # ex: 30000
debug = conf.getboolean("metrics", "otel_debugging_on")
resource = Resource(attributes={SERVICE_NAME: "Airflow"})
protocol = "https" if ssl_active else "http"
endpoint = f"{protocol}://{host}:{port}{path}"
```
### Use case/motivation
Ability to send metrics directly into OTEL-compatible metrics collectors, e.g. VictoriaMetrics.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in airflow/metrics/otel_logger.py at get_otel_logger and review the existing metrics configuration entries used there. Add a configurable OTEL path with /v1/metrics as the default, then verify that endpoints can target the example collector paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100