prometheus-community / prometheus-community/stackdriver_exporter

Error in scraping bigtable.googleapis.com/server/latencies

Open
#67 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
117
Avg merge
9d 14h
Merged PRs (30d)
7

Description

This started failing around the 17th Dec 2019 - it was running fine before that date.

We run the exporter using

/opt/stackdriver_exporter/stackdriver_exporter --google.project-id <snipped> --monitoring.metrics-type-prefixes "bigtable.googleapis.com,logging.googleapis.com,compute.googleapis.com/instance/uptime" --stackdriver.http-timeout=45s

We ran it in debug mode, and this is the error it fails with

DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/server/returned_rows_count`...  source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/server/sent_bytes_count`...  source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/disk_load`...  source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/cpu_load`...  source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/cpu_load_hottest_node`...  source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/storage_utilization`...  source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/disk/bytes_used`...  source="monitoring_collector.go:172"
ERRO[0030] Error retrieving Time Series metrics for descriptor `bigtable.googleapis.com/server/latencies`: googleapi: Error 500: Failed to convert server response to JSON, backendError  source="monitoring_collector.go:182"
ERRO[0030] Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 500: Failed to convert server response to JSON, backendError  source="monitoring_collector.go:138"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied stackdriver_exporter command with the Bigtable metric prefix and inspect monitoring_collector.go around lines 138, 172, and 182. Check how the exporter handles the server/latencies descriptor and the Google Monitoring response; done means that descriptor no longer causes the collection to fail with the JSON conversion error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.