hashicorp / hashicorp/consul

Global ACL tokens are required for Consul Connect when using local mesh gateways

Open
#7,381 4 comments 6 reactions 0 assignees View on GitHub
needs-discussion type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

A sidecar proxy using a local ACL token will not be able to route to another dc's connect service even if mesh gateways are in local mode.

The following will be logged on the Consul client:
```
2020-03-03T17:28:18.005Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.244.2.30:8300 error="rpc error making call: rpc error making call: ACL not found"
2020-03-03T17:28:18.005Z [ERROR] agent.proxycfg: watch error: id=upstream-target:static-server.default.dc2:static-server?dc=dc2 error="error filling agent cache: rpc error making call: rpc error making call: ACL not found"
2020-03-03T17:28:18.066Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.244.2.30:8300 error="rpc error making call: rpc error making call: ACL not found"
```

This is a particular issue in Kubernetes because we use `consul login` to create our tokens and this always returns a local token.

#### Reproduction Steps
1. Create two datacenters with acls enabled and federate them
1. Set a proxy-defaults config:
```
"kind": "proxy-defaults",
"name": "global",
"mesh_gateway": {
"mode": "local
}
```
1. Start mesh gateways in both dcs
1. Create a local ACL token in dc1
1. Start a sidecar proxy in dc1 using the local ACL token with an upstream of a service in dc2
1. You should see the errors in the logs

#### Suggested Solution
We should short-circuit where we iterate over the upstreams and start blocking queries: https://github.com/hashicorp/consul/blob/e83fb1882c26d8d3dd1715efa5b88654be059f86/agent/proxycfg/state.go#L263-L317.

Instead we should check if we're using local gateways and not make these calls. The results of these calls are discarded later if using local mesh gateways so we don't need them.

#### Future
In order to use mesh gateways in remote mode or to use tokens from `consul login` to make cross-dc calls, we need another solution. We could make the `consul login` tokens global, but then login would require the primary DC to be available. A better long-term solution would be to federate trust such that locally minted tokens can be trusted globally.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.