prometheus-community / prometheus-community/stackdriver_exporter

Timeout fetching Google stackdriver monitoring metrics (Client.Timeout exceeded while awaiting headers)

Open
#152 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
117
Avg merge
9d 14h
Merged PRs (30d)
7

Description

Issue

Stackdriver exporter throws the below error when fetching stack driver monitoring metrics.

{ labels: { compute.googleapis.com/resource_name: "gke-company-resource-name" k8s-pod/app: "prometheus-stackdriver-exporter" k8s-pod/release: "stable" } logName: "projects/company/logs/stderr" receiveTimestamp: "2022-04-20T01:15:51.291425040Z" resource: { labels: { cluster_name: "company-cluster1" container_name: "prometheus-stackdriver-exporter" namespace_name: "internal-services" pod_name: "stable-prometheus-stackdriver-exporter-pod-suffix" project_id: "company-project-id" } type: "k8s_container" } severity: "ERROR" textPayload: "level=error ts=2022-04-20T01:15:49.913Z caller=monitoring_collector.go:185 msg="Error while getting Google Stackdriver Monitoring metrics" err="Get \"https://monitoring.googleapis.com/v3/projects/company-cluster1/metricDescriptors?alt=json&filter=metric.type+%3D+starts_with%28%22compute.googleapis.com%2Finstance%2Fcpu%22%29&prettyPrint=false\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"" timestamp: "2022-04-20T01:15:49.913872755Z" }

Question

What is the timeout parameter that has to increased to prevent timing out ? Below Helm chart is used to deploy stack driver exporter.

image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in monitoring_collector.go around line 185 and inspect the Helm chart used to deploy the exporter. Trace the Google Cloud Monitoring client timeout configuration and determine which setting controls the request shown in the error. Done means the responsible timeout parameter is identified and its configuration or documentation addresses the reported timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud, prometheus
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.