auto_config vs. enable_token_replication in secondary datacenters
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
When configuring auto_config on a secondary datacenter (federated setup wit mesh gateways), that will not work because token replication is not enabled in secondary datacenters.
Consul Version:
root@consul-0:~# consul version
Consul v1.17.3
Revision 009041f8
Build Date 2024-02-13T18:30:29Z
---
#### Reproduction Steps
1. Set up a federated consul setup with a primary and at least a secondary datacenter. This involves activating ACLs and token replication
2. Try to activate auto_config on the Consul servers in the secondary datacenter
3. Consul will not start
4. `consul validate` will produce this error:
```
root@consul-0:/etc/consul# consul validate consul.hcl
Config validation failed: Enabling auto-config authorization (auto_config.authorization.enabled) in non primary datacenters
with ACLs enabled (acl.enabled) requires also enabling ACL token replication (acl.enable_token_replication)
```
### Consul info for both Client and Server
consul info from server:
```
root@consul-0:~# consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 009041f8
version = 1.17.3
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 4
leader = false
leader_addr = 192.168.0.211:8300
server = true
raft:
applied_index = 607130
commit_index = 607130
fsm_pending = 0
last_contact = 49.871139ms
last_log_index = 607130
last_log_term = 119
last_snapshot_index = 606392
last_snapshot_term = 119
latest_configuration = [{Suffrage:Voter ID:03e09c98-016e-8899-9e82-3e3d69078069 Address:192.168.0.211:8300} {Suffrage:Voter ID:9f6f1ef9-67fc-ce4a-0653-3e2ec5119e13 Address:192.168.0.88:8300} {Suffrage:Voter ID:fd5d2820-7e0f-d9bf-d096-41e1e6662235 Address:192.168.0.66:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 119
runtime:
arch = amd64
cpu_count = 2
goroutines = 240
max_procs = 2
os = linux
version = go1.21.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 48
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 164
members = 11
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 920
members = 12
query_queue = 0
query_time = 1
```
consul.hcl from server
```
datacenter = "dc1"
primary_datacenter = "dc2" # is needed on primaries _and_ secondaries
data_dir = "/opt/consul"
log_level = "INFO"
node_name = "consul-0"
server = true
bootstrap_expect = 3
retry_join = [redacted]
ports {
http = -1
https = 8501
grpc = 8502
grpc_tls = 8503
}
ui_config {
enabled = true
}
encrypt = "sometoken"
bind_addr = "0.0.0.0"
advertise_addr = "{{ GetInterfaceIP \"ens3\" }}"
advertise_addr_wan = "someaddress"
client_addr = "0.0.0.0"
translate_wan_addrs = true
alt_domain = "prod1"
dns_config {
enable_truncate = true
udp_answer_limit = 100
}
recursors = ["someaddress","someotheraddress"]
ca_file = "/etc/consul/certificates/ca.pem"
cert_file = "/etc/consul/certificates/cert.pem"
key_file = "/etc/consul/certificates/private_key.pem"
verify_incoming = false
verify_outgoing = false
verify_server_hostname = false
acl {
enabled = true
default_policy = "deny"
enable_token_persistence = true # persists tokens to disk
# enable_token_replication = true # is only needed on primary
down_policy = "extend-cache"
tokens {
master = "anothertoken"
agent = "yetanothertoken"
replication = "replicationtoken" # is only needed on secondaries
}
}
primary_gateways = [ "gatewayone:8443","gatewaytwo:8443" ]
connect {
enabled = true
enable_mesh_gateway_wan_federation = true
}
config_entries {
bootstrap = [
{
kind = "proxy-defaults"
name = "global"
config {
protocol = "http"
envoy_prometheus_bind_addr = "0.0.0.0:9102"
}
mesh_gateway = {
mode = "local"
}
}
]
}
auto_config {
authorization {
enabled = true
static {
jwt_validation_pub_keys = ["-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYPzTTT8YVv64JUJ8mSNe4n6WdYZak+dj1yvPh6ILqBcKhXgslasgefDLm8EjGVfFA8pppBrrm+2DDHy52WRarA==\n-----END PUBLIC KEY-----\n"]
bound_issuer = "secint"
bound_audiences = ["dc1"]
claim_mappings {
sub = "node_name"
}
claim_assertions = [
"value.node_name == \"${node}\""
]
}
}
}
performance {
raft_multiplier = 1
}
```
### Operating system and Environment details
This is on Linux - Ubuntu 22.04
### The story so far
- I have tried to activate token replication on the secondary - that leads to `ACL not found` errors and no longer being able to login
- The docs (https://developer.hashicorp.com/consul/docs/security/acl/acl-federated-datacenters?productSlug=consul&tutorialSlug=security-operations&tutorialSlug=access-control-replication-multiple-datacenters) explicitly say, that enable_token_replication is only needed on the primary
- Is auto_config no longer possible in federated setups?
Contributor guide
Research direction
Start with the auto_config.authorization.enabled and acl.enable_token_replication validation paths, using the reported consul validate command and the federated datacenter configuration as the reproduction. Compare the documented primary and secondary ACL settings, then establish whether the configuration should be accepted or the documentation and validation behavior should be aligned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- authentication, distributed-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100