prometheus-community / prometheus-community/stackdriver_exporter
too many buckets in Cloud Run metrics
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 295
- Forks
- 117
- Avg merge
- 9d 14h
- Merged PRs (30d)
- 7
Description
I started using Cloud Run and added run.googleapis.com to the prefixes in stackdriver_exporter
I have only ~50 runs/containers, but the exporter started publishing 150,000 metrics and when I looked at them, it turned out that 99% of them were _bucket metrics
for example:
- stackdriver_cloud_run_revision_run_googleapis_com_container_probe_attempt_latencies_bucket geneates 235 buckets per EVERY container (container_probe_attempt_latencies distribution x 235 buckets)
buckets looks like this:
[235 lines]
le="717.9517789085845"
le="789.7469567994431"
le="868.7216524793874"
le="955.5938177273262"
le="1051.153199500059"
le="1156.2685194500648"
le="1271.8953713950714"
le="1399.0849085345785"
le="1538.9933993880366"
le="1692.8927393268405"
le="1862.1820132595244"
le="2048.400214585477"
le="2253.2402360440246"
le="2478.564259648427"
le="2726.420685613271"
le="2999.0627541745976"
le="3298.9690295920577"
le="3628.865932551264"
...
- stackdriver_cloud_run_revision_run_googleapis_com_container_cpu_utilizations_bucket publishes 102 buckets per EVERY container (container_cpu_utilizations distribution x 102 buckets)
buckets looks like this:
le="0.00"
le="0.01"
....[
le="0.94"
le="0.95"
le="0.96"
le="0.97"
le="0.98"
le="0.99"
...
(every percent has it's own bucket)
how to deal with it? is it possible to customize number of retrieved buckets during querying googleapi, or maybe it's something on GCP side (some settings hard to find)?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue using the run.googleapis.com prefixes and Cloud Run distribution metrics described here, then trace how the exporter queries and publishes distribution buckets. Determine whether bucket counts can be controlled through the Google API or exporter configuration; done means the excessive _bucket series can be limited or a clear limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud, prometheus
- Domain
- cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100