open-telemetry / open-telemetry/opentelemetry-python

Opentelemetry prometheus exported datadog parser error

Open
#3,568 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Trying to expose a prometheus endpoint so that datadog can pull the custom metrics from that endpoint. For this I am using a envvironment variable OTEL_METRICS_EXPORTER=prometheus, which opens up a port at 9464 to scrap metrics.

When datadog parse the response from the endpoint :9464/metrics, the datadog agent throws exception related to parsing.

Steps to reproduce

  1. Create a Java project using opentelemetry dependencies and create a guage
  2. Try to record some metrics in the gauge.
  3. Add env variable called OTEL_METRICS_EXPORTER=prometheus
  4. Start the program
  5. Configure Datadog agent to scrap from the prometheus endpoint i.e. :9464/metrics
  6. Datadog agent should throw parsing exception

What is the expected behavior?
The custom metrics should reach to the datadog metrics's dashboard

What is the actual behavior?
Parsing error thrown by datadog agent

Additional context
Attaching
datadog_error.txt
prometheus_exported_response.txt
prometheus exported response and the parsing error from datadog.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing prometheus_exported_response.txt with datadog_error.txt and review the Prometheus exporter behavior enabled by OTEL_METRICS_EXPORTER=prometheus. Reproduce the gauge export and Datadog scrape using the six listed steps. Done means the exported response is accepted by the Datadog agent and the custom metric reaches its dashboard.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.