hashicorp / hashicorp/consul

Global management policy changes causing issue

Open
#22,116 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
A change in the global management policy is causing errors. After upgrading the primary Consul datacenter from 1.15.10 to 1.17.3, while the secondary datacenter remains on 1.15.10, we are experiencing ACL replication failures in the secondary datacenters.

The secondary DC logs show the error:
"Changing the Rules for the builtin global-management policy is not permitted."

Post-upgrade, we noticed that the global management policy ACL in the primary DC includes a new section that is missing in the secondary DC.

```
identity_prefix "" {
policy = "write"
intentions = "write"
}
```

---

#### Reproduction Steps

1. Make sure both primary DC and secondary DC is on 1.15.10
2. Upgrade primary DC from 1.15.10 to 1.17.3. Keep Secondary in 1.15.10.

### Consul info for both Client and Server

Server info - Primary DC

```
consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 009041f8
version = 1.17.3
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = <>:8300
server = true
raft:
applied_index = 218893
commit_index = 218893
fsm_pending = 0
last_contact = 110.729µs
last_log_index = 218893
last_log_term = 5
last_snapshot_index = 213165
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:11155734-3e22-e2b4-2e8a-89798ddada36 Address:<>:8300} {Suffrage:Voter ID:d71e3df6-33af-e9b0-2291-84362f68bd54 Address:<>8300} {Suffrage:Voter ID:c325fad8-29e9-b46e-6695-52d6bc849386 Address:<>:8300} {Suffrage:Voter ID:7bdd3d50-f665-f32e-3f7b-dcf50e7e7ca1 Address:<>:8300} {Suffrage:Voter ID:88462f52-4dc9-03b4-fc73-e01885f8d1ad Address:<>:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 5
runtime:
arch = amd64
cpu_count = 2
goroutines = 192
max_procs = 2
os = linux
version = go1.21.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 5
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 607
members = 12
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 423
members = 11
query_queue = 0
query_time = 1
```

```
/consul/config # cat server.json
{
"client_addr": "{{ GetInterfaceIP \"eth1\" }} 127.0.0.1",
"bind_addr": "{{ GetInterfaceIP \"eth1\" }}",
"data_dir": "/consul/data",
"log_level": "INFO",
"datacenter": "${DC}",
"encrypt": "${GOSSIP_KEY}",
"primary_datacenter": "${PRIMARY_CLUSTER}",
"retry_join_wan": ["${RETRY_DNS}:8302"],
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"leave_on_terminate": true,
"skip_leave_on_interrupt": false,
"tls": {
"defaults": {
"tls_cipher_suites": "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"verify_incoming": false,
"verify_outgoing": true,
"ca_file": "ca.cert.pem",
"cert_file": "consul.crt",
"key_file": "consul.key"
},
"internal_rpc": {
"verify_server_hostname": true,
"verify_incoming": true
}
},
"rpc": {
"enable_streaming": true
},
"ports": {
"http": -1,
"https": 8501,
"grpc": -1,
"grpc_tls": 8502
},
"auto_encrypt": {
"allow_tls": true
},
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true,
"enable_token_replication": true,
"tokens": {
"master": "${MASTER_TOKEN}",
"agent": "${REPLICATION_TOKEN}",
"replication": "${REPLICATION_TOKEN}"
}
},
"connect": {
"enabled": true
},
"telemetry": {
"statsd_address": "localhost:8125",
"disable_hostname": true
}
}
```

### Operating system and Environment details

### Log Fragments

```2025-01-27T04:42:28.386Z [WARN] agent.server.replication.acl.policy: ACL replication error (will retry if still leader): error="failed to update local ACL policies: Failed to apply policy upserts: Changing the Rules for the builtin global-management policy is not permitted" ```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the 1.15.10-to-1.17.3 upgrade described in the issue, using the provided server.json configuration and the ACL replication warning as checkpoints. Compare the global-management policy between the primary and secondary datacenters and trace the ACL replication path. Done means replication succeeds across the mixed-version datacenters without the builtin-policy error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.