hashicorp / hashicorp/consul

consul-agents metrics requests denied for anon token when default_policy = "allow"

Open
#20,807 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

consul-agent's deny requests to metrics endpoint and info when ACL is enabled despite `default_policy = "allow"`. This appears to be true for most requests made from an agent when using auto_config.

I've validated that CONSUL_HTTP_TOKEN is not set when making these requests

#### Reproduction Steps

Steps to reproduce this issue:

1. Create a cluster following this tutorial https://developer.hashicorp.com/consul/tutorials/security-operations/docker-compose-auto-config (update server config`default_policy: "allow"`)
1. shell into an agent
1. make a request, get an error:
2.
```
/ # curl localhost:8500/v1/agent/metrics?format=prometheus
Permission denied: anonymous token lacks permission 'agent:read' on "ip-10-0-10-78.ec2.internal". The anonymous token is used implicitly when a request does not specify a token./ #
```

### Consul info for both Client and Server

On the client agent this was painful to get because most commands failed with lack of acl permissions for agent:read, i wound up using the intial management token to get these

Client info

Output from client 'consul info' command here

```
agent:
check_monitors = 0
check_ttls = 0
checks = 6
services = 2
build:
prerelease =
revision = 1.16.3-s
version = 1.16.3
version_metadata =
consul:
acl = enabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 32
goroutines = 411
max_procs = 1
os = linux
version = go1.22.0
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 366
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 9322
members = 6
query_queue = 0
query_time = 42
```

Client agent HCL config
```
disable_update_check = true
telemetry {
prometheus_retention_time = "24h"
disable_hostname = true
}
limits {
http_max_conns_per_client = 512
}
tls {
https {
ca_file = "/tls/server.cacrt"
verify_incoming = false
verify_outgoing = true
}
internal_rpc {
ca_file = "/tls/server.cacrt"
verify_incoming = false
verify_outgoing = true
}
}
auto_config = {
enabled = true
intro_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
server_addresses = [ "provider=k8s namespace=consul label_selector=\"name=consul,component=server\"" ]
}
```

Server info

Output from server 'consul info' command here
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 1.16.3-s
version = 1.16.3
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.0.20.34:8300
server = true
raft:
applied_index = 681728
commit_index = 681728
fsm_pending = 0
last_contact = 0
last_log_index = 681728
last_log_term = 1173
last_snapshot_index = 672786
last_snapshot_term = 1155
latest_configuration = [{Suffrage:Voter ID:3ac501a0-d3c5-58eb-3d23-cfd1dbc06aa5 Address:10.0.20.34:8300} {Suffrage:Voter ID:6bc2b476-fb32-45cf-70f2-99bba7786e03 Address:10.0.14.215:8300} {Suffrage:Voter ID:92aa5db7-b795-a305-93aa-30ca8101df66 Address:10.0.30.32:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 1173
runtime:
arch = amd64
cpu_count = 32
goroutines = 219
max_procs = 1
os = linux
version = go1.22.0
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 366
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 9322
members = 6
query_queue = 0
query_time = 42
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 4960
members = 3
query_queue = 0
query_time = 2
```

Server agent HCL config
```
cat /consul/config/server.hcl
data_dir = "/consul/data"

domain = "consul"

recursors = []

tls = {
https = {
verify_incoming = true
verify_outgoing = true
cert_file = "/tls/server.crt"
key_file = "/tls/server.key"
ca_file = "/tls/server.cacrt"
}

internal_rpc = {
verify_incoming = true
verify_outgoing = true
cert_file = "/tls/server.crt"
key_file = "/tls/server.key"
ca_file = "/tls/server.cacrt"
}
}

acl = {
enabled = true
default_policy = "allow"
}

server = true

bootstrap_expect = 3

disable_remote_exec = true

enable_debug = true

performance = {
raft_multiplier = 1
}

raft_boltdb = {
NoFreeListSync = true
}

disable_update_check = true

disable_host_node_id = true

telemetry = {
prometheus_retention_time = "24h"
disable_hostname = true
}

dns_config = {
service_ttl = {
brubeck = "10s"
c2cvault = "10s"
graphite-pickle-relay = "10s"
graphite-plaintext-relay = "10s"
proxyout = "5s"
}

use_cache = true
allow_stale = true
cache_max_age = "10s"
node_ttl = "1m"

soa = {
min_ttl = 5
}

enable_additional_node_meta_txt = false
}
connect = {
enabled = true
ca_provider = "vault"

# Note: these are only for bootstrapping: https://developer.hashicorp.com/consul/docs/agent/config/config-files#connect_ca_config
ca_config = {
address = "..."
root_pki_path = "service-to-service-root"
intermediate_pki_path = "consul-connect"
intermediate_cert_ttl = "720h"
csr_max_per_second = 100

auth_method = {
type = "aws"
}
}
}
auto_config {
authorization {
enabled = true
static {
#jwks_url = "http://127.0.0.1:40008/keys" # with or without path- seems like it needs path
oidc_discovery_url = "..." # TODO enter the k8s oidc provider
bound_issuer = "..." # TODO k8s oidc provider
bound_audiences = ["sts.amazonaws.com"] # TODO IDK
claim_mappings = {
"/kubernetes.io/namespace" = "namespace",
"/kubernetes.io/pod/name" = "pod",
"/kubernetes.io/serviceaccount/name" = "serviceaccountname"
}
# claim_mappings = { # TODO IDK
#consul/hostname: node_name
#sub = "system:serviceaccount:kube-system:consul-agent"
#}
}
}
}

client_addr = "0.0.0.0"

ports = {
grpc = 8502
http = 8500
}
datacenter = "sndbox-ue1-aws"

primary_datacenter = "sndbox-ue1-aws"

retry_join = [
#"provider=aws tag_key=consulserver tag_value=sndbox-ue1",
"provider=k8s namespace=consul label_selector=\"name=consul,component=server\""
]
enable_central_service_config = true
```

### Operating system and Environment details

this is based of official consul 1.16.3 image running in containers on aws

### Log Fragments

Contributor guide

Open the contributing guide

Research direction

Start at the /v1/agent/metrics endpoint and the agent request authorization path, then trace how auto_config requests are authenticated when no CONSUL_HTTP_TOKEN is set. Reproduce the denial with default_policy = "allow" and confirm that anonymous metrics and info requests receive the expected access under that policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
api, authorization, backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.