hashicorp / hashicorp/consul

[middleware] turn off noisy RPC call observations

Open
#12,814 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

### overview

A complex consul cluster will make various RPC calls for internal operations. For instance, `Status.RaftStats` or `Coordinate.Update`. Unfortunately, these calls do not actually show a lot of value and they can cause more noise in the metrics than offer actual signals.

### proposal

We need to find a way to "filter out" RPC calls that we don't really want to be observed, for whatever reason. Ideally, this is a dynamic or "reloadable" operation, in that it's not going to require restarting all Consul server.

Let's look at a couple proposals.

#### P1

Label based filtering is supported by `go-metrics`. However, our `consul.rpc.service.call` metric looks like this:

```
{
"metric": consul.rpc.service.call,
"label": value
...
}
```

If we were to filter out a label, then all labels get filtered out. We only want to filter out the label when its value matches `== Coordinate.Update` (for example).

That is why we may want to implement https://github.com/armon/go-metrics/issues/131,

This proposal is the recommended way to solve this since it supports dynamic filtering reload.

#### P2

It may be that implementing P1 is too heavy, or, indeed, that implementing it in `go-metrics` does not make sense.

Then the layer for us to implement at would the `RequestRecorder`. We would proably need to add:

* config support for which label values we don't want to support
* add interceptor style to any `go-metrics` call to take out the filtered our label values (or just start with the `RequestRecorder`
* make said config hot reloadable

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the RequestRecorder and the consul.rpc.service.call metric, then compare the proposed go-metrics label-filtering approach with the alternative configuration layer. The issue points to go-metrics issue 131 for P1. Done should provide configurable filtering of selected RPC label values with dynamic reload support, without restarting Consul servers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.