hashicorp / hashicorp/consul

Metric labels rewriting for statsd sink

Open
#8,250 0 comments 0 reactions 0 assignees View on GitHub
theme/telemetry type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Feature Description

Metric labels are passed as-is to go-metrics where they are handled differently depending on the implementation. In the case of the statsd reporter, labels are joined into one string with a dot as a separator creating a hierarchical type metric in Graphite. This works well for most of the cases but it gets complicated when a label itself contains a dot, e.g. a hostname or an IP address.

It would be great to have an option to sanitize or rewrite labels.

#### Use Case(s)

Labels like this can be observed in metrics like [`consul.client.rpc.failed`](https://github.com/hashicorp/consul/blob/abce1f0eeead917adef3caa3a4d8e3e1f227255c/agent/consul/client.go#L335) and [`consul.catalog.service.query-tag`](https://github.com/hashicorp/consul/blob/a7f69b615a192892d9c3c60626baa32c7d7078b5/agent/consul/catalog_endpoint.go#L475).

Such behavior makes it hard to properly aggregate metrics, especially when single metric gets split into multiple metrics with different levels in hierarchy due to label length, e.g.

`consul.client.rpc.failed.consul1.local.count`
`consul.client.rpc.failed.consul2.local.count`
`consul.client.rpc.failed.consul1.cloud.azure.count`
which creates the following hierarchy
```
consul
- client
- rpc
- failed
- consul1
- local
- count
- cloud
- azure
- count
- consul2
- local
- count
```
As you can see `count` in `local` and `cloud.azure` are on a different level.

If dots in labels could be rewritten to some other character, like an underscore `_`, the example metric above would have the same depth of hierarchy.

`consul.client.rpc.failed.consul1_local.count`
`consul.client.rpc.failed.consul1_cloud_azure.count`
which results in this hierarchy:
```
consul
- client
- rpc
- failed
- consul1_local
- count
- consul1_cloud_azure
- count
- consul2_local
- count
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.