Global management policy changes causing issue
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
A change in the global management policy is causing errors. After upgrading the primary Consul datacenter from 1.15.10 to 1.17.3, while the secondary datacenter remains on 1.15.10, we are experiencing ACL replication failures in the secondary datacenters.
The secondary DC logs show the error:
"Changing the Rules for the builtin global-management policy is not permitted."
Post-upgrade, we noticed that the global management policy ACL in the primary DC includes a new section that is missing in the secondary DC.
```
identity_prefix "" {
policy = "write"
intentions = "write"
}
```
---
#### Reproduction Steps
1. Make sure both primary DC and secondary DC is on 1.15.10
2. Upgrade primary DC from 1.15.10 to 1.17.3. Keep Secondary in 1.15.10.
### Consul info for both Client and Server
Server info - Primary DC
```
consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 009041f8
version = 1.17.3
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = <>:8300
server = true
raft:
applied_index = 218893
commit_index = 218893
fsm_pending = 0
last_contact = 110.729µs
last_log_index = 218893
last_log_term = 5
last_snapshot_index = 213165
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:11155734-3e22-e2b4-2e8a-89798ddada36 Address:<>:8300} {Suffrage:Voter ID:d71e3df6-33af-e9b0-2291-84362f68bd54 Address:<>8300} {Suffrage:Voter ID:c325fad8-29e9-b46e-6695-52d6bc849386 Address:<>:8300} {Suffrage:Voter ID:7bdd3d50-f665-f32e-3f7b-dcf50e7e7ca1 Address:<>:8300} {Suffrage:Voter ID:88462f52-4dc9-03b4-fc73-e01885f8d1ad Address:<>:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 5
runtime:
arch = amd64
cpu_count = 2
goroutines = 192
max_procs = 2
os = linux
version = go1.21.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 5
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 607
members = 12
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 423
members = 11
query_queue = 0
query_time = 1
```
```
/consul/config # cat server.json
{
"client_addr": "{{ GetInterfaceIP \"eth1\" }} 127.0.0.1",
"bind_addr": "{{ GetInterfaceIP \"eth1\" }}",
"data_dir": "/consul/data",
"log_level": "INFO",
"datacenter": "${DC}",
"encrypt": "${GOSSIP_KEY}",
"primary_datacenter": "${PRIMARY_CLUSTER}",
"retry_join_wan": ["${RETRY_DNS}:8302"],
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"leave_on_terminate": true,
"skip_leave_on_interrupt": false,
"tls": {
"defaults": {
"tls_cipher_suites": "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"verify_incoming": false,
"verify_outgoing": true,
"ca_file": "ca.cert.pem",
"cert_file": "consul.crt",
"key_file": "consul.key"
},
"internal_rpc": {
"verify_server_hostname": true,
"verify_incoming": true
}
},
"rpc": {
"enable_streaming": true
},
"ports": {
"http": -1,
"https": 8501,
"grpc": -1,
"grpc_tls": 8502
},
"auto_encrypt": {
"allow_tls": true
},
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true,
"enable_token_replication": true,
"tokens": {
"master": "${MASTER_TOKEN}",
"agent": "${REPLICATION_TOKEN}",
"replication": "${REPLICATION_TOKEN}"
}
},
"connect": {
"enabled": true
},
"telemetry": {
"statsd_address": "localhost:8125",
"disable_hostname": true
}
}
```
### Operating system and Environment details
### Log Fragments
```2025-01-27T04:42:28.386Z [WARN] agent.server.replication.acl.policy: ACL replication error (will retry if still leader): error="failed to update local ACL policies: Failed to apply policy upserts: Changing the Rules for the builtin global-management policy is not permitted" ```
Contributor guide
Research direction
Start by reproducing the 1.15.10-to-1.17.3 upgrade described in the issue, using the provided server.json configuration and the ACL replication warning as checkpoints. Compare the global-management policy between the primary and secondary datacenters and trace the ACL replication path. Done means replication succeeds across the mixed-version datacenters without the builtin-policy error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100