prometheus-community / prometheus-community/stackdriver_exporter
Error in scraping bigtable.googleapis.com/server/latencies
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 295
- Forks
- 117
- Avg merge
- 9d 14h
- Merged PRs (30d)
- 7
Description
This started failing around the 17th Dec 2019 - it was running fine before that date.
We run the exporter using
/opt/stackdriver_exporter/stackdriver_exporter --google.project-id <snipped> --monitoring.metrics-type-prefixes "bigtable.googleapis.com,logging.googleapis.com,compute.googleapis.com/instance/uptime" --stackdriver.http-timeout=45s
We ran it in debug mode, and this is the error it fails with
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/server/returned_rows_count`... source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/server/sent_bytes_count`... source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/disk_load`... source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/cpu_load`... source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/cpu_load_hottest_node`... source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/cluster/storage_utilization`... source="monitoring_collector.go:172"
DEBU[0030] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigtable.googleapis.com/disk/bytes_used`... source="monitoring_collector.go:172"
ERRO[0030] Error retrieving Time Series metrics for descriptor `bigtable.googleapis.com/server/latencies`: googleapi: Error 500: Failed to convert server response to JSON, backendError source="monitoring_collector.go:182"
ERRO[0030] Error while getting Google Stackdriver Monitoring metrics: googleapi: Error 500: Failed to convert server response to JSON, backendError source="monitoring_collector.go:138"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied stackdriver_exporter command with the Bigtable metric prefix and inspect monitoring_collector.go around lines 138, 172, and 182. Check how the exporter handles the server/latencies descriptor and the Google Monitoring response; done means that descriptor no longer causes the collection to fail with the JSON conversion error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100