prometheus-community / prometheus-community/stackdriver_exporter

appengine.googleapis.com causes errors in prometheus handler

Open
#38 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
117
Avg merge
9d 14h
Merged PRs (30d)
7

Description

Hello,

When we include appengine.googleapis.com in our prefix list we see 500s being returned from the /metrics endpoint. We've turned debug logging on and it seems to be loading the app engine metrics without error:

time="2018-10-18T19:02:01Z" level=debug msg="Listing Google Stackdriver Monitoring metric descriptors starting with `appengine.googleapis.com`..." source="monitoring_collector.go:210"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/system/network/sent_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/http/server/quota_denial_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/system/memory/usage`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/system/network/received_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/http/server/response_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/http/server/dos_intercept_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/flex/disk/write_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/http/server/response_latencies`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/memcache/centi_mcu_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/memcache/operation_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/memcache/received_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/memcache/sent_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/system/cpu/usage`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/system/instance_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/flex/cpu/reserved_cores`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/flex/cpu/utilization`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/flex/disk/read_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/flex/network/sent_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/flex/network/received_bytes_count`..." source="monitoring_collector.go:166"
time="2018-10-18T19:02:01Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `appengine.googleapis.com/http/server/response_style_count`..." source="monitoring_collector.go:166"

There are no error logs coming from the prometheus handler (or any error logs coming in general) and I can't seem to find a way to turn on more detailed logging for the handler (new to Go so may be missing something).

Any idea what could be causing this?

Thanks for a great project, we use this in many of our Kubernetes deployments and its been very helpful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the /metrics handler and the retrieval path in monitoring_collector.go, especially the code around the logged lines 166 and 210. Reproduce the 500 response with appengine.googleapis.com in the prefix list, then identify the failing metric or response conversion and add coverage showing that the handler no longer returns an error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud, prometheus
Domain
cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.