prometheus-community / prometheus-community/stackdriver_exporter

Apigee metrics from GCP - bucket type prometheus metrics value is not increasing monotonically

Open
#336 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
117
Avg merge
9d 14h
Merged PRs (30d)
7

Description

We've been introduced to this stackdriver_exporter recently and started using it to export metrics from GCP infra to our prometheus. Specifically the metrics of Apigee proxy provided as a managed service in GCP. We found a specific type of metric which got exported through the stackdriver exporter is not increasing monotonically in its value over the time. This metric type is used to bucket the range of values in different buckets which then can be used to calculate quantiles using histogram_quantile function with prometheus.
Here is an example for metric which is used for calculating latency whose value didn't increase monotonically

test

Error got from prometheus while calculating histogram quantile using the above metric

test1

This is a helm based deployment in a kubernetes cluster.

Helm Chart version deployed - v4.5.0
Exporter app version deployed - v0.15.0

Expected:
The metric _bucket value has to be increasing always like a counter type metric

Help needed . Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is identified in the report. Start by reproducing the Apigee _bucket metric behavior with Helm chart v4.5.0 and exporter v0.15.0, then trace how the metric is exported and compare it with Prometheus histogram expectations. Done means the bucket values increase monotonically and histogram_quantile no longer reports the shown error.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go, helm, kubernetes, prometheus
Domain
cloud, infrastructure, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.