Modify envoy stats_matcher to reduce envoy prometheus metrics

Background

The default configuration for Envoy's statistics is pretty broad and includes a TON of metrics. With over 200+ instances of envoy, we find prometheus is using a lot of memory and there's a fair bit of network traffic just for scraping these metrics.

When I look around in the community, I've found that istio limits what stats they expose in envoy in the default configuration (they have docs here: https://istio.io/latest/docs/ops/configuration/telemetry/envoy-stats/)

I looked to see if there was a way to perform a similar filtering of metrics collectors for our envoys through consul, and it doesn't appear to be any way to configure envoy StatsMatcher. The StatsMatcher configuration allows for filtering the set of metrics exposed by a whitelist or blacklist via pattern matching.

Feature Request

My feature request is the ability to disable envoy metrics collectors via consul configuration.

I think it would be valuable to have a global setting for configuring StatsMatchers, and also the ability to override on a per-instance basis (i.e. via pod annotations) for exceptions where you want an additional set of metrics etc.

Use Case(s)

I'm almost certain that by default a number of metrics exposed by envoy are never really viewed/queried/used by users. Turning these off by default in consul configuration is desirable to reduce prometheus resource usage and network bandwidth usage.

No source files, tests, or entry points are identified in the issue. Start by tracing how Consul configures Envoy instances and whether StatsMatcher can be represented globally and through per-instance annotations. Done means users can configure metric filtering globally and override it for individual instances, with verification that the resulting Envoy metrics are filtered.