Agent telemetry filter_default configuration option not respected by prometheus interface
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
When calling the agent telemetry endpoint /v1/agent/metrics, metrics can be viewed by default in a json format or using the parameter format=prometheus to receive the metrics in prometheus format. By default, all metrics described in the [documentation](https://developer.hashicorp.com/consul/docs/agent/monitor/telemetry) are available in both json and prometheus format.
However, if the [prefix_filter](https://developer.hashicorp.com/consul/docs/agent/config/config-files#telemetry-prefix_filter) option is set, the configuration seems to only apply to the non-Prometheus view of the metrics. Similarly, [filter_default](https://developer.hashicorp.com/consul/docs/agent/config/config-files#telemetry-filter_default) does not have any effect on the prometheus view of metrics.
#### Reproduction Steps
1. Start an agent with no prefix_filter parameter
```
consul agent -dev -node localhost -client 127.0.0.1 -hcl 'telemetry { prometheus_retention_time = "10m" }'
```
2. Check for a metric such as consul.serf in both outputs
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics | head
{
"Timestamp": "2024-10-15 23:02:40 +0000 UTC",
"Gauges": [
{
"Name": "consul.302com1.autopilot.failure_tolerance",
"Value": 0,
"Labels": {}
},
{
"Name": "consul.302com1.autopilot.healthy",
```
```
root@mynode:/state/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics?format=prometheus | head
# HELP consul_302com1_autopilot_failure_tolerance consul_302com1_autopilot_failure_tolerance
# TYPE consul_302com1_autopilot_failure_tolerance gauge
consul_302com1_autopilot_failure_tolerance 0
# HELP consul_302com1_autopilot_healthy consul_302com1_autopilot_healthy
# TYPE consul_302com1_autopilot_healthy gauge
consul_302com1_autopilot_healthy 1
# HELP consul_302com1_cache_entries_count consul_302com1_cache_entries_count
# TYPE consul_302com1_cache_entries_count gauge
consul_302com1_cache_entries_count 1
```
3. Start an agent with a prefix_filter parameter such as removing `consul.serf` metrics
```consul agent -dev -node localhost -client 127.0.0.1 -hcl 'telemetry { prometheus_retention_time = "10m", filter_default = false, prefix_filter = ["+consul.serf"] }'```
4. Confirm the configuration is in place on the agent
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/self | jq -r '.DebugConfig.Telemetry'
{
"AllowedPrefixes": [],
"BlockedPrefixes": [
"consul.serf",
"consul.rpc.server.call"
],
...
"EnableHostMetrics": false,
"FilterDefault": false,
"MetricsPrefix": "consul",
```
5. Check metrics on both the json and prometheus metrics interface to see that serf metrics are the only ones remaining on the non-prometheus result but prometheus still contains other metrics
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics | head
{
"Timestamp": "2024-10-15 23:19:00 +0000 UTC",
"Gauges": [],
"Points": [],
"Counters": [],
"Samples": [
{
"Name": "consul.serf.queue.Event",
"Count": 1,
"Rate": 0.1,
```
```
root@mynode:/home/wolfmd# curl -sS 127.0.0.1:8500/v1/agent/metrics?format=prometheus | head
# HELP consul_acl_ResolveToken This measures the time it takes to resolve an ACL token.
# TYPE consul_acl_ResolveToken summary
consul_acl_ResolveToken{quantile="0.5"} NaN
consul_acl_ResolveToken{quantile="0.9"} NaN
consul_acl_ResolveToken{quantile="0.99"} NaN
consul_acl_ResolveToken_sum 0
consul_acl_ResolveToken_count 0
# HELP consul_acl_authmethod_delete
# TYPE consul_acl_authmethod_delete summary
consul_acl_authmethod_delete{quantile="0.5"} NaN
```
### Consul info for both Client and Server
Agent is running consul 1.17.4. This can be reproduced in agent dev mode
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease = dev
revision = 3e2302b+
version = 1.17.4
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 127.0.0.1:8300
server = true
raft:
applied_index = 64
commit_index = 64
fsm_pending = 0
last_contact = 0
last_log_index = 64
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:27ef875d-74af-30ff-1c7e-0ed5b987609b Address:127.0.0.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 96
goroutines = 186
max_procs = 96
os = linux
version = go1.22.5 X:boringcrypto
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
### Operating system and Environment details
Running on bare metal Debian
Contributor guide
Research direction
Start at the /v1/agent/metrics endpoint handling and trace how telemetry filtering is applied for JSON versus Prometheus output. Reproduce the issue with filter_default=false and prefix_filter using the commands shown, then verify that both formats honor the configured filters and add regression coverage for the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- api, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100