hashicorp / hashicorp/consul

Agent telemetry filter_default configuration option not respected by prometheus interface

Open
#21,831 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

When calling the agent telemetry endpoint /v1/agent/metrics, metrics can be viewed by default in a json format or using the parameter format=prometheus to receive the metrics in prometheus format. By default, all metrics described in the [documentation](https://developer.hashicorp.com/consul/docs/agent/monitor/telemetry) are available in both json and prometheus format.

However, if the [prefix_filter](https://developer.hashicorp.com/consul/docs/agent/config/config-files#telemetry-prefix_filter) option is set, the configuration seems to only apply to the non-Prometheus view of the metrics. Similarly, [filter_default](https://developer.hashicorp.com/consul/docs/agent/config/config-files#telemetry-filter_default) does not have any effect on the prometheus view of metrics.

#### Reproduction Steps
1. Start an agent with no prefix_filter parameter
```
consul agent -dev -node localhost -client 127.0.0.1 -hcl 'telemetry { prometheus_retention_time = "10m" }'
```

2. Check for a metric such as consul.serf in both outputs
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics | head
{
"Timestamp": "2024-10-15 23:02:40 +0000 UTC",
"Gauges": [
{
"Name": "consul.302com1.autopilot.failure_tolerance",
"Value": 0,
"Labels": {}
},
{
"Name": "consul.302com1.autopilot.healthy",
```
```
root@mynode:/state/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics?format=prometheus | head
# HELP consul_302com1_autopilot_failure_tolerance consul_302com1_autopilot_failure_tolerance
# TYPE consul_302com1_autopilot_failure_tolerance gauge
consul_302com1_autopilot_failure_tolerance 0
# HELP consul_302com1_autopilot_healthy consul_302com1_autopilot_healthy
# TYPE consul_302com1_autopilot_healthy gauge
consul_302com1_autopilot_healthy 1
# HELP consul_302com1_cache_entries_count consul_302com1_cache_entries_count
# TYPE consul_302com1_cache_entries_count gauge
consul_302com1_cache_entries_count 1
```

3. Start an agent with a prefix_filter parameter such as removing `consul.serf` metrics
```consul agent -dev -node localhost -client 127.0.0.1 -hcl 'telemetry { prometheus_retention_time = "10m", filter_default = false, prefix_filter = ["+consul.serf"] }'```

4. Confirm the configuration is in place on the agent
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/self | jq -r '.DebugConfig.Telemetry'
{
"AllowedPrefixes": [],
"BlockedPrefixes": [
"consul.serf",
"consul.rpc.server.call"
],
...
"EnableHostMetrics": false,
"FilterDefault": false,
"MetricsPrefix": "consul",
```

5. Check metrics on both the json and prometheus metrics interface to see that serf metrics are the only ones remaining on the non-prometheus result but prometheus still contains other metrics
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics | head
{
"Timestamp": "2024-10-15 23:19:00 +0000 UTC",
"Gauges": [],
"Points": [],
"Counters": [],
"Samples": [
{
"Name": "consul.serf.queue.Event",
"Count": 1,
"Rate": 0.1,
```
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics?format=prometheus | head
# HELP consul_acl_ResolveToken This measures the time it takes to resolve an ACL token.
# TYPE consul_acl_ResolveToken summary
consul_acl_ResolveToken{quantile="0.5"} NaN
consul_acl_ResolveToken{quantile="0.9"} NaN
consul_acl_ResolveToken{quantile="0.99"} NaN
consul_acl_ResolveToken_sum 0
consul_acl_ResolveToken_count 0
# HELP consul_acl_authmethod_delete
# TYPE consul_acl_authmethod_delete summary
consul_acl_authmethod_delete{quantile="0.5"} NaN
```

### Consul info for both Client and Server
Agent is running consul 1.17.4. This can be reproduced in agent dev mode
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease = dev
revision = 3e2302b+
version = 1.17.4
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 127.0.0.1:8300
server = true
raft:
applied_index = 64
commit_index = 64
fsm_pending = 0
last_contact = 0
last_log_index = 64
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:27ef875d-74af-30ff-1c7e-0ed5b987609b Address:127.0.0.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 96
goroutines = 186
max_procs = 96
os = linux
version = go1.22.5 X:boringcrypto
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Running on bare metal Debian

Contributor guide

Open the contributing guide

Research direction

Start at the /v1/agent/metrics endpoint handling and trace how telemetry filtering is applied for JSON versus Prometheus output. Reproduce the issue with filter_default=false and prefix_filter using the commands shown, then verify that both formats honor the configured filters and add regression coverage for the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
api, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.