ACL System crashes on RHEL on VMWare servers
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
We decided to upgrade our infrastructure slowly from 1.9.1 to 1.11. Our servers reside on VMWare Infrastructure, based on RHEL servers.
Every time we tried migrating, the ACL system decided to crash, and the master token became invalid, while we can't view the status of our services and nodes. We also tried to reset the ACL System as showed in the outage troubleshooting, to no success.
We reproduced it by booting a new, seperate Consul cluster, and it showed the same result the moment we booted up the ACL System.
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Create a cluster, using Consul version 1.9.3 or above.
2. Use the following configuration files:
`config.acl`:
```
{
"bind_addr": ">>bind ip<<",
"addresses": {
"dns": "0.0.0.0",
"http": "0.0.0.0",
"https": "0.0.0.0"
},
"ports": {
"dns": 53,
"http": 8500,
"https": 8501,
"serf_lan": 8301,
"serf_wan": 8302,
"server": 8300
},
"bootstrap": true,
"server": true,
"node_name": "consul-node1",
"datacenter": "prod",
"data_dir": "/var/consul",
"encrypt": ">>key<<",
"log_level": "INFO",
"enable_syslog": true,
"domain": "consul",
"dns_config": {
"allow_stale": "true",
"max_stale": "30s",
"service_ttl": {
"*": "5s"
},
"node_ttl": "5s",
"only_passing": true
},
"ui_config": {
"enabled": true
}
}
```
`config_acl.json`:
```
{
"primary_datacenter": "prod",
"acl": {
"enabled": true,
"down_policy": "deny",
"default_policy": "deny",
"tokens": {
"master": ">>master token<<",
"agent_master": ">>master token",
"default": "anonymous"
}
}
}
```
3. The log shows either the error `ACL not found` or `Permission denied` while trying to connect with the master token. Access to services and nodes that registered with Consul is permitted if the anonymous token has read access over the services and nodes on the cluster, but the UI won't show them.
### Operating system and Environment details
OS: Red Hat Enterprise Linux 7.4 (although the bug reproduced in 8.x as well)
The servers reside on an on-prem VMWare infrastructure.
Contributor guide
Research direction
Start by reproducing the issue with Consul 1.9.3 or later using the supplied config.acl and config_acl.json files on RHEL, then inspect logs for “ACL not found” or “Permission denied.” Done means the ACL system remains valid after startup, the master token works, and the UI can show registered services and nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- authorization, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100