prometheus-community / prometheus-community/stackdriver_exporter
Metrics gap due to "Treat failure to collect metric as fatal"
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 295
- Forks
- 117
- Avg merge
- 9d 14h
- Merged PRs (30d)
- 7
Description
We're experiencing small gaps in the metrics exported by the stackdriver exporter and, after some investigation, ended up to be caused by the stackdriver exporter exiting each time an API request timeout:
level=error ts=2020-06-30T12:54:18.862Z caller=monitoring_collector.go:194 msg="Error while getting Google Stackdriver Monitoring metrics" err="Get \"REDACTED\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
This behavioural change has been introduced in https://github.com/prometheus-community/stackdriver_exporter/pull/83 and I'm wondering how we could improve it. Few ideas/options:
- Revert PR #83
- Exit with fatal error only for specific errors (in this case I would need to better understand the specific error returned by the #66 use case)
- Add a CLI flag to change the "exit on metric collection error" behaviour, to be able to enable/disable it
Any other idea to fix it?
Example of gap

Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with monitoring_collector.go at line 194 and review the behavior introduced by pull request #83, then compare it with the #66 use case. Investigate how request timeouts are handled and decide whether fatal handling should be reverted, narrowed, or made configurable; done requires an agreed behavior that prevents the reported metric gaps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100