consul-agents metrics requests denied for anon token when default_policy = "allow"
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
consul-agent's deny requests to metrics endpoint and info when ACL is enabled despite `default_policy = "allow"`. This appears to be true for most requests made from an agent when using auto_config.
I've validated that CONSUL_HTTP_TOKEN is not set when making these requests
#### Reproduction Steps
Steps to reproduce this issue:
1. Create a cluster following this tutorial https://developer.hashicorp.com/consul/tutorials/security-operations/docker-compose-auto-config (update server config`default_policy: "allow"`)
1. shell into an agent
1. make a request, get an error:
2.
```
/ # curl localhost:8500/v1/agent/metrics?format=prometheus
Permission denied: anonymous token lacks permission 'agent:read' on "ip-10-0-10-78.ec2.internal". The anonymous token is used implicitly when a request does not specify a token./ #
```
### Consul info for both Client and Server
On the client agent this was painful to get because most commands failed with lack of acl permissions for agent:read, i wound up using the intial management token to get these
Client info
Output from client 'consul info' command here
```
agent:
check_monitors = 0
check_ttls = 0
checks = 6
services = 2
build:
prerelease =
revision = 1.16.3-s
version = 1.16.3
version_metadata =
consul:
acl = enabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 32
goroutines = 411
max_procs = 1
os = linux
version = go1.22.0
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 366
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 9322
members = 6
query_queue = 0
query_time = 42
```
Client agent HCL config
```
disable_update_check = true
telemetry {
prometheus_retention_time = "24h"
disable_hostname = true
}
limits {
http_max_conns_per_client = 512
}
tls {
https {
ca_file = "/tls/server.cacrt"
verify_incoming = false
verify_outgoing = true
}
internal_rpc {
ca_file = "/tls/server.cacrt"
verify_incoming = false
verify_outgoing = true
}
}
auto_config = {
enabled = true
intro_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
server_addresses = [ "provider=k8s namespace=consul label_selector=\"name=consul,component=server\"" ]
}
```
Server info
Output from server 'consul info' command here
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 1.16.3-s
version = 1.16.3
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.0.20.34:8300
server = true
raft:
applied_index = 681728
commit_index = 681728
fsm_pending = 0
last_contact = 0
last_log_index = 681728
last_log_term = 1173
last_snapshot_index = 672786
last_snapshot_term = 1155
latest_configuration = [{Suffrage:Voter ID:3ac501a0-d3c5-58eb-3d23-cfd1dbc06aa5 Address:10.0.20.34:8300} {Suffrage:Voter ID:6bc2b476-fb32-45cf-70f2-99bba7786e03 Address:10.0.14.215:8300} {Suffrage:Voter ID:92aa5db7-b795-a305-93aa-30ca8101df66 Address:10.0.30.32:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 1173
runtime:
arch = amd64
cpu_count = 32
goroutines = 219
max_procs = 1
os = linux
version = go1.22.0
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 366
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 9322
members = 6
query_queue = 0
query_time = 42
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 4960
members = 3
query_queue = 0
query_time = 2
```
Server agent HCL config
```
cat /consul/config/server.hcl
data_dir = "/consul/data"
domain = "consul"
recursors = []
tls = {
https = {
verify_incoming = true
verify_outgoing = true
cert_file = "/tls/server.crt"
key_file = "/tls/server.key"
ca_file = "/tls/server.cacrt"
}
internal_rpc = {
verify_incoming = true
verify_outgoing = true
cert_file = "/tls/server.crt"
key_file = "/tls/server.key"
ca_file = "/tls/server.cacrt"
}
}
acl = {
enabled = true
default_policy = "allow"
}
server = true
bootstrap_expect = 3
disable_remote_exec = true
enable_debug = true
performance = {
raft_multiplier = 1
}
raft_boltdb = {
NoFreeListSync = true
}
disable_update_check = true
disable_host_node_id = true
telemetry = {
prometheus_retention_time = "24h"
disable_hostname = true
}
dns_config = {
service_ttl = {
brubeck = "10s"
c2cvault = "10s"
graphite-pickle-relay = "10s"
graphite-plaintext-relay = "10s"
proxyout = "5s"
}
use_cache = true
allow_stale = true
cache_max_age = "10s"
node_ttl = "1m"
soa = {
min_ttl = 5
}
enable_additional_node_meta_txt = false
}
connect = {
enabled = true
ca_provider = "vault"
# Note: these are only for bootstrapping: https://developer.hashicorp.com/consul/docs/agent/config/config-files#connect_ca_config
ca_config = {
address = "..."
root_pki_path = "service-to-service-root"
intermediate_pki_path = "consul-connect"
intermediate_cert_ttl = "720h"
csr_max_per_second = 100
auth_method = {
type = "aws"
}
}
}
auto_config {
authorization {
enabled = true
static {
#jwks_url = "http://127.0.0.1:40008/keys" # with or without path- seems like it needs path
oidc_discovery_url = "..." # TODO enter the k8s oidc provider
bound_issuer = "..." # TODO k8s oidc provider
bound_audiences = ["sts.amazonaws.com"] # TODO IDK
claim_mappings = {
"/kubernetes.io/namespace" = "namespace",
"/kubernetes.io/pod/name" = "pod",
"/kubernetes.io/serviceaccount/name" = "serviceaccountname"
}
# claim_mappings = { # TODO IDK
#consul/hostname: node_name
#sub = "system:serviceaccount:kube-system:consul-agent"
#}
}
}
}
client_addr = "0.0.0.0"
ports = {
grpc = 8502
http = 8500
}
datacenter = "sndbox-ue1-aws"
primary_datacenter = "sndbox-ue1-aws"
retry_join = [
#"provider=aws tag_key=consulserver tag_value=sndbox-ue1",
"provider=k8s namespace=consul label_selector=\"name=consul,component=server\""
]
enable_central_service_config = true
```
### Operating system and Environment details
this is based of official consul 1.16.3 image running in containers on aws
### Log Fragments
Contributor guide
Research direction
Start at the /v1/agent/metrics endpoint and the agent request authorization path, then trace how auto_config requests are authenticated when no CONSUL_HTTP_TOKEN is set. Reproduce the denial with default_policy = "allow" and confirm that anonymous metrics and info requests receive the expected access under that policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- api, authorization, backend, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100