GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver
Overwrites MonitoredDescriptor, dropping all historical data
- Dominant language
- Go
- Stars
- 409
- Forks
- 236
- Avg merge
- 2h 34m
- Merged PRs (30d)
- 9
Description
Prometheus-to-sd tries to update the Stackdriver MonitoredDescriptor if the installed one is incompatible with the Prometheus metric. This is a lossy operation that will drop all the historical data, and it could be triggered by unexpected changes or bugs in the Prometheus exporters.
I propose that we refuse the write and log an error. The metric descriptor update should be pulled out into a separate tool that requires manual invocation, for users who are OK losing the data.
The code in question is here: https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd/translator/stackdriver.go#L88
MonitoredDescriptor.Create is documented here, I've asked the owners to document this behavior explicitly: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/create
h/t to quentin
Contributor guide
Assessment
This issue has not been assessed yet.