hashicorp / hashicorp/consul

ACLs in cluster broken when installed by helm

Open
#21,805 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

When installing a 3-node cluster via `helm`, and setting `manageSystemACLs: true`, anonymous tokens are used and produce errors.

---

#### Reproduction Steps

`helm install --set acls.manageSystemACLs=true --set server.replicas=3`

Observe logs in the resulting pods:
```
2024-10-03T19:21:51.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:35520 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:22:06.593Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:49148 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:22:21.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:33454 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2
```

-->

### Consul info for both Client and Server

N/A, everything is defaulted from the chart

### Operating system and Environment details

OKD 4.17 K8s cluster, bone-stock.

### Log Fragments

```
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxyconfiguration
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinations
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/httproute
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/exportedservices
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/telemetrystate
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/workload
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/partitiontrafficpermissions
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/concept
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/artist
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/tcproute
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinationpolicy
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshconfiguration
2024-10-03T19:18:58.799Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2024-10-03T19:18:58.799Z [INFO] agent: Started gRPC listeners: port_name=grpc address=[::]:8502 network=tcp
2024-10-03T19:18:58.799Z [INFO] agent: started state syncer
2024-10-03T19:18:58.799Z [INFO] agent: Consul agent running!
2024-10-03T19:18:58.799Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2024-10-03T19:18:58.799Z [INFO] agent: Joining cluster...: cluster=LAN
2024-10-03T19:18:58.799Z [INFO] agent: (LAN) joining: lan_addresses=["consul-consul-server.consul.svc:8301"]
2024-10-03T19:18:58.844Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-consul-server-2 10.131.0.126
2024-10-03T19:18:58.845Z [INFO] agent.server: Adding LAN server: server="consul-consul-server-2 (Addr: tcp/10.131.0.126:8300) (DC: mgmt-0001)"
2024-10-03T19:18:58.847Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-consul-server-2.mgmt-0001 10.131.0.126
2024-10-03T19:18:58.847Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-consul-server-2.mgmt-0001 area=wan
2024-10-03T19:18:58.893Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-consul-server-1 10.129.2.120
2024-10-03T19:18:58.894Z [INFO] agent.server: Adding LAN server: server="consul-consul-server-1 (Addr: tcp/10.129.2.120:8300) (DC: mgmt-0001)"
2024-10-03T19:18:58.894Z [INFO] agent: (LAN) joined: number_of_nodes=3
2024-10-03T19:18:58.894Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
2024-10-03T19:18:58.897Z [INFO] agent.server: Found expected number of peers, attempting bootstrap: peers="10.128.2.123:8300,10.131.0.126:8300,10.129.2.120:8300"
2024-10-03T19:18:58.901Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-consul-server-1.mgmt-0001 10.129.2.120
2024-10-03T19:18:58.901Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-consul-server-1.mgmt-0001 area=wan
2024-10-03T19:19:06.136Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
2024-10-03T19:19:07.548Z [INFO] agent.server: New leader elected: payload=consul-consul-server-2
2024-10-03T19:19:08.242Z [WARN] agent: Node info update blocked by ACLs: node=e5639b5f-3d15-2792-59dd-af68731a1b31 accessorID="anonymous token"
2024-10-03T19:19:10.073Z [INFO] agent: Updated agent's ACL token: token=agent
2024-10-03T19:19:12.969Z [INFO] agent: Synced node info
2024-10-03T19:19:14.594Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:37500 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:19:29.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:35436 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:19:44.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:54786 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:19:59.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:42874 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:20:14.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:34498 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the provided Helm install command and inspect the resulting server pod logs. Trace the chart's ACL and token configuration to determine why requests to /v1/agent/self use the anonymous token; done means a three-node installation with manageSystemACLs enabled no longer produces the reported permission errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes
Domain
authorization, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.