Export metrics samples as Histograms rather than Summaries
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Feature Description
Currently, Consul exports the timing samples it takes as prometheus Summaries. Summaries directly quantiles of the measurements, making them useful at a glance and saving the metrics backend from performing this aggregation. However, summaries have a major limit in that quantiles do not compose across services. Instead of pre-computing their quantiles, histograms place the samples into buckets, which count their occurrences. This allows users to sum histogram buckets across many services before computing the quantiles from these aggregated measurements. Given that Consul servers are clustered by default - histograms provide a much better fit for Consul's timing measurements than prometheus summaries. In exchange, there's a higher learning curve and label required from operators - quantiles much be intentionally calculated by users.
Go-metrics provides an API for timing Samples, which is the same unit that composes into either Summaries or Histograms. The fact that we use prometheus' summary API is a small implementation detail. We call go-metrics' `AddSample` api, and this is aggregated into a local summary.
As a result, one option we have to implement Histograms in Consul is to add the storage of Histograms in the `PrometheusSink` and provide a configuration option that allows users to chose between whether Consul's samples are stored as histograms or summaries. It *is* possible to export both - but summaries and histograms are both high-cardinality instruments, each exporting on the order of 5-10 separate metrics to be composed into the respective instrument. Choosing between one or the other is better for our users so we do not overwhelm their timeseries stores.
#### Use Case(s)
While histograms have been investigated in the context of Prometheus, it could be useful in other contexts, such as the InmemSink.
Contributor guide
Assessment
This issue has not been assessed yet.