prometheus-community / prometheus-community/stackdriver_exporter

Metrics gap due to "Treat failure to collect metric as fatal"

Open
#101 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
295
Forks
117
Avg merge
9d 14h
Merged PRs (30d)
7

Description

We're experiencing small gaps in the metrics exported by the stackdriver exporter and, after some investigation, ended up to be caused by the stackdriver exporter exiting each time an API request timeout:

level=error ts=2020-06-30T12:54:18.862Z caller=monitoring_collector.go:194 msg="Error while getting Google Stackdriver Monitoring metrics" err="Get \"REDACTED\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

This behavioural change has been introduced in https://github.com/prometheus-community/stackdriver_exporter/pull/83 and I'm wondering how we could improve it. Few ideas/options:

  1. Revert PR #83
  2. Exit with fatal error only for specific errors (in this case I would need to better understand the specific error returned by the #66 use case)
  3. Add a CLI flag to change the "exit on metric collection error" behaviour, to be able to enable/disable it

Any other idea to fix it?

Example of gap

Screen Shot 2020-06-30 at 17 36 52

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with monitoring_collector.go at line 194 and review the behavior introduced by pull request #83, then compare it with the #66 use case. Investigate how request timeouts are handled and decide whether fatal handling should be reverted, narrowed, or made configurable; done requires an agreed behavior that prevents the reported metric gaps.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.