GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver
Metric descriptor validation should only look at metric fields and ignore monitored resource fields?
- Dominant language
- Go
- Stars
- 409
- Forks
- 236
- Avg merge
- 2h 34m
- Merged PRs (30d)
- 9
Description
I ran into a validation failure when trying to export a metric like: [kubernetes.io/container/accelerator/duty_cycle](https://cloud.google.com/monitoring/api/metrics_kubernetes) to stackdriver. My component emits the following prometheus metric:
```
duty_cycle{accelerator_id="gpu0",container="test-container",make="nvidia",model="nvidia-tesla-a100",namespace="test-namespace",pod="test-pod"} 7
```
I've configured prometheus-to-sd to run as follows:
```
command:
- /monitor
- --stackdriver-prefix=kubernetes.io/container
- --monitored-resource-type-prefix=k8s_
- --source=accelerator:http://localhost:2112?whitelisted=duty_cycle&namespaceIdLabel=namespace&podIdLabel=pod&containerNameLabel=container
```
However this fails with the following error (from prometheus-to-sd logs):
```
I1006 05:45:45.816338 1 metric_descriptor_cache.go:166] Missing label &{ container []} in the original metric descriptor
W1006 05:45:45.816380 1 metric_descriptor_cache.go:88] Definition of the metric duty_cycle was changed and metric is not going to be pushed
I1006 05:45:45.816389 1 translator.go:199] Monitored process start time: 2020-10-06 05:43:43 +0000 UTC
I1006 05:45:45.816401 1 translator.go:210] Exporting only whitelisted metrics: [duty_cycle]
I1006 05:45:45.816409 1 stackdriver.go:39] No metrics to send to Stackdriver for component
```
The error seems to originate from [here](https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd/translator/metric_descriptor_cache.go#L166). It complains that the metric I'm trying to export contains the container label, which is missing from the original metric descriptor definition. This looks like a bug to me because container is configured to be a monitored resource label (in the --source flag), so this will not be a metric label when publishing to SD ([reference](https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd/translator/translator.go#L306-L314)).
To fix this, should we only be looking at metric labels when validating if labels have changed [here](https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd/translator/metric_descriptor_cache.go#L156)?
Contributor guide
Assessment
This issue has not been assessed yet.