Metric Names contain duplicated words when configured to use Datadog
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
The `/v1/agent/metrics` endpoint returns metrics with duplicated sections, such as:
```
{
"Name": "consul.autopilot.failure_tolerance.failure_tolerance",
"Value": 0,
"Labels": {}
},
{
"Name": "consul.autopilot.healthy.healthy",
"Value": 1,
"Labels": {}
},
{
"Name": "consul.consul.state.nodes.nodes",
"Value": 1,
"Labels": {
"datacenter": "dc1"
}
},
```
Note the `failure_tolerance.failure_tolerance` / `healthy.healthy` / `nodes.nodes` endings.
This also impacts the other metric sinks, such as Prometheus, etc.
#### Reproduction Steps
1. Configure Consul telemetry with Datadog sink
```
"telemetry": {
"dogstatsd_addr": "127.0.0.1:8125",
"prometheus_retention_time": "300s"
},
```
2. Make sure `disable_hostname` is `false`. https://www.consul.io/docs/agent/options#telemetry-disable_hostname
2. Query the `/v1/agent/metrics` endpoint
#### Cause
I believe the issue is caused by the Datadog sink reusing an array to [scrap off the hostname](https://github.com/armon/go-metrics/blob/8ca742bd545671bedae063d0672cb3c8da7d4437/datadog/dogstatsd.go#L73). This causes other sinks, that receive the same `key []string` slice, to record the duplicated endings. Here is a Go Playground snippet showing this behavior: https://play.golang.org/p/JcfbFU0otS3
```go
package main
import (
"fmt"
)
func main() {
fmt.Println("Hello, playground")
gauge := []string{"hostname", "consul", "state", "nodes"}
newG := parseKey(gauge)
fmt.Printf("old (%p %#v) new (%p %#v)\n", gauge, gauge, newG, newG)
}
func parseKey(key []string) []string {
for i, el := range key {
if el == "hostname" {
key = append(key[:i], key[i+1:]...)
break
}
}
return key
}
```
Contributor guide
Assessment
This issue has not been assessed yet.