prometheus-community / prometheus-community/stackdriver_exporter

too many buckets in Cloud Run metrics

Open
#438 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
117
Avg merge
9d 14h
Merged PRs (30d)
7

Description

I started using Cloud Run and added run.googleapis.com to the prefixes in stackdriver_exporter

I have only ~50 runs/containers, but the exporter started publishing 150,000 metrics and when I looked at them, it turned out that 99% of them were _bucket metrics

for example:

  • stackdriver_cloud_run_revision_run_googleapis_com_container_probe_attempt_latencies_bucket geneates 235 buckets per EVERY container (container_probe_attempt_latencies distribution x 235 buckets)

buckets looks like this:

[235 lines]
le="717.9517789085845"
le="789.7469567994431"
le="868.7216524793874"
le="955.5938177273262"
le="1051.153199500059"
le="1156.2685194500648"
le="1271.8953713950714"
le="1399.0849085345785"
le="1538.9933993880366"
le="1692.8927393268405"
le="1862.1820132595244"
le="2048.400214585477"
le="2253.2402360440246"
le="2478.564259648427"
le="2726.420685613271"
le="2999.0627541745976"
le="3298.9690295920577"
le="3628.865932551264"
...
  • stackdriver_cloud_run_revision_run_googleapis_com_container_cpu_utilizations_bucket publishes 102 buckets per EVERY container (container_cpu_utilizations distribution x 102 buckets)

buckets looks like this:

le="0.00"
le="0.01"
....[
le="0.94"
le="0.95"
le="0.96"
le="0.97"
le="0.98"
le="0.99"
...

(every percent has it's own bucket)

how to deal with it? is it possible to customize number of retrieved buckets during querying googleapi, or maybe it's something on GCP side (some settings hard to find)?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue using the run.googleapis.com prefixes and Cloud Run distribution metrics described here, then trace how the exporter queries and publishes distribution buckets. Determine whether bucket counts can be controlled through the Google API or exporter configuration; done means the excessive _bucket series can be limited or a clear limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud, prometheus
Domain
cloud, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.