GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

Overwrites MonitoredDescriptor, dropping all historical data

Open
#69 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

Prometheus-to-sd tries to update the Stackdriver MonitoredDescriptor if the installed one is incompatible with the Prometheus metric. This is a lossy operation that will drop all the historical data, and it could be triggered by unexpected changes or bugs in the Prometheus exporters.

I propose that we refuse the write and log an error. The metric descriptor update should be pulled out into a separate tool that requires manual invocation, for users who are OK losing the data.

The code in question is here: https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd/translator/stackdriver.go#L88

MonitoredDescriptor.Create is documented here, I've asked the owners to document this behavior explicitly: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/create

h/t to quentin

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.