hashicorp / hashicorp/consul

ACL System crashes on RHEL on VMWare servers

Open
#14,108 0 comments 0 reactions 0 assignees View on GitHub
theme/acls type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

We decided to upgrade our infrastructure slowly from 1.9.1 to 1.11. Our servers reside on VMWare Infrastructure, based on RHEL servers.
Every time we tried migrating, the ACL system decided to crash, and the master token became invalid, while we can't view the status of our services and nodes. We also tried to reset the ACL System as showed in the outage troubleshooting, to no success.

We reproduced it by booting a new, seperate Consul cluster, and it showed the same result the moment we booted up the ACL System.

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a cluster, using Consul version 1.9.3 or above.
2. Use the following configuration files:
`config.acl`:
```
{
"bind_addr": ">>bind ip<<",
"addresses": {
"dns": "0.0.0.0",
"http": "0.0.0.0",
"https": "0.0.0.0"
},
"ports": {
"dns": 53,
"http": 8500,
"https": 8501,
"serf_lan": 8301,
"serf_wan": 8302,
"server": 8300
},
"bootstrap": true,
"server": true,
"node_name": "consul-node1",
"datacenter": "prod",
"data_dir": "/var/consul",
"encrypt": ">>key<<",
"log_level": "INFO",
"enable_syslog": true,
"domain": "consul",
"dns_config": {
"allow_stale": "true",
"max_stale": "30s",
"service_ttl": {
"*": "5s"
},
"node_ttl": "5s",
"only_passing": true
},
"ui_config": {
"enabled": true
}
}
```

`config_acl.json`:
```
{
"primary_datacenter": "prod",
"acl": {
"enabled": true,
"down_policy": "deny",
"default_policy": "deny",
"tokens": {
"master": ">>master token<<",
"agent_master": ">>master token",
"default": "anonymous"
}
}
}
```
3. The log shows either the error `ACL not found` or `Permission denied` while trying to connect with the master token. Access to services and nodes that registered with Consul is permitted if the anonymous token has read access over the services and nodes on the cluster, but the UI won't show them.

### Operating system and Environment details

OS: Red Hat Enterprise Linux 7.4 (although the bug reproduced in 8.x as well)
The servers reside on an on-prem VMWare infrastructure.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with Consul 1.9.3 or later using the supplied config.acl and config_acl.json files on RHEL, then inspect logs for “ACL not found” or “Permission denied.” Done means the ACL system remains valid after startup, the master token works, and the UI can show registered services and nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
authorization, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.