[middleware] turn off noisy RPC call observations
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
### overview
A complex consul cluster will make various RPC calls for internal operations. For instance, `Status.RaftStats` or `Coordinate.Update`. Unfortunately, these calls do not actually show a lot of value and they can cause more noise in the metrics than offer actual signals.
### proposal
We need to find a way to "filter out" RPC calls that we don't really want to be observed, for whatever reason. Ideally, this is a dynamic or "reloadable" operation, in that it's not going to require restarting all Consul server.
Let's look at a couple proposals.
#### P1
Label based filtering is supported by `go-metrics`. However, our `consul.rpc.service.call` metric looks like this:
```
{
"metric": consul.rpc.service.call,
"label": value
...
}
```
If we were to filter out a label, then all labels get filtered out. We only want to filter out the label when its value matches `== Coordinate.Update` (for example).
That is why we may want to implement https://github.com/armon/go-metrics/issues/131,
This proposal is the recommended way to solve this since it supports dynamic filtering reload.
#### P2
It may be that implementing P1 is too heavy, or, indeed, that implementing it in `go-metrics` does not make sense.
Then the layer for us to implement at would the `RequestRecorder`. We would proably need to add:
* config support for which label values we don't want to support
* add interceptor style to any `go-metrics` call to take out the filtered our label values (or just start with the `RequestRecorder`
* make said config hot reloadable
Contributor guide
Research direction
Start by reviewing the RequestRecorder and the consul.rpc.service.call metric, then compare the proposed go-metrics label-filtering approach with the alternative configuration layer. The issue points to go-metrics issue 131 for P1. Done should provide configurable filtering of selected RPC label values with dynamic reload support, without restarting Consul servers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100