ACL Replication error="ACL not found" after 1.7.4 upgrade
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
#### Overview of the Issue
I am currently running a 3 region federated cluster with 5 nodes in each region. `us-east-1`, `us-west-2`, `eu-central-1`. The cluster has replicated ACLs with `us-east-1` as the `primary_datacenter`. The cluster was running `Consul v1.6.1` and I am in the process of upgrading to `Consul v1.7.4`.
`us-east-1` servers were upgraded completely without issue or error. I moved to upgrade the first server in `us-west-2` and am receiving the repeated error `error="ACL not found"` for various calls now on the first WAN node in `us-west-2`, even using the `global-management` token.
The newly upgrdaded node `consul.usw2` has rejoined the cluster and wan. The remaining nodes which haven't been upgraded yet are still functioning properly.
### Consul info for both Client and Server
Server consul-n01.use1 info
```
root@consul-n01:~# consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = d149d7e9
version = 1.7.4
consul:
acl = enabled
bootstrap = false
known_datacenters = 3
leader = false
leader_addr = 10.2.27.169:8300
server = true
raft:
applied_index = 7572552
commit_index = 7572552
fsm_pending = 0
last_contact = 41.515046ms
last_log_index = 7572552
last_log_term = 1808
last_snapshot_index = 7572354
last_snapshot_term = 1808
latest_configuration = [{Suffrage:Voter ID:b54b7910-856d-200e-232c-09f64ccd4df3 Address:10.2.30.107:8300} {Suffrage:Voter ID:1d5cb5ee-91df-ea6a-8882-ae9b776503bc Address:10.2.36.43:8300} {Suffrage:Voter ID:5679dad8-a54e-7b66-c412-fd0ff8a1762d Address:10.2.27.169:8300} {Suffrage:Voter ID:0d512eba-6de7-e558-43a2-32e2c7db7e0e Address:10.2.51.37:8300} {Suffrage:Voter ID:f990f4c4-4b5e-0127-f07a-1fb6d4aa3ec0 Address:10.2.39.52:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 1808
runtime:
arch = amd64
cpu_count = 2
goroutines = 135
max_procs = 2
os = linux
version = go1.13.12
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 85
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1199
members = 23
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 343
members = 15
query_queue = 0
query_time = 1
```
Server consul-n01.usw2 info
```
root@consul-n01:/etc/consul.d# consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = d149d7e9
version = 1.7.4
consul:
acl = enabled
bootstrap = false
known_datacenters = 3
leader = false
leader_addr = 10.3.49.45:8300
server = true
raft:
applied_index = 1712953
commit_index = 1712953
fsm_pending = 0
last_contact = 36.740742ms
last_log_index = 1712953
last_log_term = 73
last_snapshot_index = 1704529
last_snapshot_term = 73
latest_configuration = [{Suffrage:Voter ID:d1f00b73-c300-8c86-1312-681bfafb1fe0 Address:10.3.21.213:8300} {Suffrage:Voter ID:dabebc0f-4716-6726-2762-edd36ec91e21 Address:10.3.25.224:8300} {Suffrage:Voter ID:743289a9-6fa3-514a-7372-572beb3844b9 Address:10.3.49.45:8300} {Suffrage:Voter ID:630d6c0b-1d59-976f-e3c9-dc74695ad04f Address:10.3.33.177:8300} {Suffrage:Voter ID:1048e780-9285-f0a4-cac3-de7b4bd13d99 Address:10.3.62.158:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 73
runtime:
arch = amd64
cpu_count = 1
goroutines = 117
max_procs = 1
os = linux
version = go1.13.12
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 34
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 305
members = 23
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 343
members = 15
query_queue = 0
query_time = 1
```
### Operating system and Environment details
```
root@consul-n01:/etc/consul.d# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
root@consul-n01:/etc/consul.d# uname -ra
Linux consul-n01 4.4.0-1057-aws #66-Ubuntu SMP Thu May 3 12:49:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```
### Log Fragments
Relevant [Configurations and logs](https://gist.github.com/powellchristoph/ac9bc01785fcde8011e7b614eea3d16f)
The two main items that appear to be the most relevant are:
* missing `TaggedAddresses` in the `consul catalog nodes -detailed` output
* and the `"ReplicationType": "policies"` in the replication output
Contributor guide
Assessment
This issue has not been assessed yet.