open-telemetry / open-telemetry/opentelemetry-python

Help on how to add labels to metrics sent to Prometheus from automatic instrumentation

Open
#3,690 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Hi, I instrumented my Flask app as you can see below (2nd code), using the automatic instrumentation and pushing the metrics to an AWS Managed Prometheus service. The setup works, I was able to visualize the metrics in Grafana, but they don't have the necessary labels to identify them, like the route (ex: "api/post"), request parameters, headers. I was able to add labels by using the Resource class ( provider.resource = Resource.create(resource_attributes)), but the labels set here will be added to every single metric and will not be request-specific. I want labels to identify the requests and be able to filter them in Grafana by the endpoint for example.
I also was able to add a specific label for service_name using an environment variable (OTEL_SERVICE_NAME=my-service).

I saw that all the attributes are already available in the instrumentation module during the requests, both in the "attributes" and "flask_request_environ" below:

def _wrapped_before_request(
    request_hook=None,
    tracer=None,
    excluded_urls=None,
    enable_commenter=True,
    commenter_options=None,
):
    def _before_request():
        if excluded_urls and excluded_urls.url_disabled(flask.request.url):
            return
        flask_request_environ = flask.request.environ
        span_name = get_default_span_name()

        attributes = otel_wsgi.collect_request_attributes(
            flask_request_environ
        )
        if flask.request.url_rule:
            # For 404 that result from no route found, etc, we
            # don't have a url_rule.
            attributes[SpanAttributes.HTTP_ROUTE] = flask.request.url_rule.rule
        span, token = _start_internal_or_server_span(
            tracer=tracer,
            span_name=span_name,
            start_time=flask_request_environ.get(_ENVIRON_STARTTIME_KEY),
            context_carrier=flask_request_environ,
            context_getter=otel_wsgi.wsgi_getter,
            attributes=attributes,
        )

I tried so many different things at this point and I really couldn't crack how to make this work. So my question is, would it be possible to use the already available labels and add them to the metrics being sent to Prometheus? That would help me a lot and I would be unbelievably grateful if someone could help me with that. Thank you very much!

def create_app():
    app = Flask(__name__)
    
    ...

    from opentelemetry.metrics import set_meter_provider
    from opentelemetry.sdk.metrics import MeterProvider
    from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
    from opentelemetry.exporter.prometheus_remote_write import (
    PrometheusRemoteWriteMetricsExporter,
     )
    from requests_auth_aws_sigv4 import AWSSigV4

    aws_auth = AWSSigV4('aps',
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
        region='eu-west-1',
    )
    from opentelemetry.sdk.resources import Resource
    exporter = PrometheusRemoteWriteMetricsExporter(
        endpoint=PROMETHEUS_REMOTE_WRITE_ENDPOINT,
        headers={},
        auth=aws_auth, 
    )

    resource = Resource.create()
    reader = PeriodicExportingMetricReader(exporter, 1000)
    provider = MeterProvider(metric_readers=[reader], resource=resource)
    set_meter_provider(provider)
    FlaskInstrumentor().instrument_app(app, meter_provider=provider)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with FlaskInstrumentor and its _wrapped_before_request entry point, then inspect the MeterProvider and PrometheusRemoteWriteMetricsExporter setup shown in the report. Determine whether request-specific attributes can be emitted as Prometheus metric labels, and document the supported approach or the missing implementation needed to filter metrics by endpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, flask, prometheus, python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.