influxdata / influxdata/prometheus_metric_normalizer

Concerns regarding out-of-the-box behavior which joins measurements with different tags

Open
#1 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Regarding using `prometheus_metric_normalizer` to pre-join `kube-state-metrics` data...

We've noticed that the default tasks are VERY broad, eg all "kube_pod*" metrics are joined, whether they have the same tags or not -- so we end up with only half a dozen measurements, sure, but 90% of the fields are blank in each row.

As a very targeted example, using the joined "kube_service" measurement, the raw data that it's trying to merge is along these lines:

```
kube_service_info{namespace="default",service="kube-state-metrics"} 1
kube_service_labels{label_app="kube-state-metrics",namespace="default",service="kube-state-metrics"} 1
```

But it doesn't merge those two entries into the same series -- instead it results in two series, one with label_app empty, the other with it populated, b/c the tags aren't identical.

eg

time | info | instance | label_app | labels | namespace | service
------------------------ | ---- | ------------------- | -------------------- | ------ | --------- | --------------------
2017-05-31T16:17:37.644Z | | "192.168.0.6:31441" | "kube-state-metrics" | 1 | "default" | "kube-state-metrics"
2017-05-31T16:17:37.644Z | 1 | "192.168.0.6:31441" | | | "default" | "kube-state-metrics"

That's just with two measurements that don't share the same tags -- with the "kube_pod" example task in `prometheus_metric_normalizer`, this effect is far more pronounced.

As a result, if we use the "kube_pod" task (for instance) out-of-the-box, then we need to re-mine the joined data in order to separate the values with data that's actually relevant to any given query... it seems like this defeats the purpose?

So our main question is -- Were the `prometheus_metric_normalizer` tasks intended to be used out-of-the-box, or were they intended as examples of how to join measurements?

If they're intended as examples, then would you be interested in some of the more targeted measurement joins that we iron out in order to facilitate our grafana graphs / kapacitor alerts? (I'm thinking we're not the only ones going down this path.)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.