"no auto-encrypt server addresses available for use" when client tries to cloud auto-join on AKS with auto-encrypt enabled
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
I am following the guide at https://www.consul.io/docs/k8s/deployment-configurations/single-dc-multi-k8s to configure a single Consul DC that will have clients join from multiple K8s clusters. The guide covers how to deploy Consul server nodes into one K8s cluster, then have clients deployed into a 2nd K8s cluster connect to the servers via cloud auto-join.
When the client starts up in the secondary cluster, it shows the following output:
```
[ERROR] agent.auto_config: no auto-encrypt server addresses available for use
```
Complete log fragment is included below.
This error message could be due to auto-encrypt, which the guide suggest to enable. This error message is found in `auto_encrypt.go` (https://github.com/hashicorp/consul/blob/v1.13.1/agent/auto-config/auto_encrypt.go#L85-L106). My reading of that code is that the error is raised if there are no `start_join` addresses provided. The Helm chart is specifying the `-retry-join` parameter to the Consul agent rather than `-start-join`.
The following is the exec statement for the Consul agent container:
```
exec /usr/local/bin/docker-entrypoint.sh consul agent \
-node="${NODE}" \
-advertise="${ADVERTISE_IP}" \
-bind=0.0.0.0 \
-client=0.0.0.0 \
-node-meta=host-ip:${HOST_IP} \
-node-meta=pod-name:${HOSTNAME} \
-hcl='leave_on_terminate = true' \
-hcl='ca_file = "/consul/tls/ca/tls.crt"' \
-hcl='auto_encrypt = {tls = true}' \
-hcl="auto_encrypt = {ip_san = [\"$HOST_IP\",\"$POD_IP\"]}" \
-hcl='verify_outgoing = true' \
-hcl='ports { https = 8501 }' \
-hcl='ports { http = -1 }' \
-hcl='ports { grpc = 8502 }' \
-config-dir=/consul/config \
-config-dir=/consul/aclconfig \
-datacenter=aks01 \
-data-dir=/consul/data \
-encrypt="${GOSSIP_KEY}" \
-retry-join="provider=k8s kubeconfig=/consul/userconfig/mgmt-kubeconfig/kubeconfig label_selector=\"app=consul,component=server\"" \
-config-file=/consul/extra-config/extra-from-values.json \
-domain=consul
```
### Consul info for both Client and Server
Client info
```
/ $ consul info
Error querying agent: Get "https://localhost:8501/v1/agent/self": dial tcp [::1]:8501: connect: connection refused
```
Server info
```
/ $ consul info -token
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = c6d0f9ec
version = 1.13.1
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.121.16.62:8300
server = true
raft:
applied_index = 73349
commit_index = 73349
fsm_pending = 0
last_contact = 3.206454ms
last_log_index = 73349
last_log_term = 74
last_snapshot_index = 65560
last_snapshot_term = 74
latest_configuration = [{Suffrage:Voter ID:b2128539-6a8d-ec3a-7c95-519f1d8949ac Address:10.121.16.105:8300} {Suffrage:Voter ID:b608ced7-5cfd-520d-071e-9a0f952e0008 Address:10.121.16.62:8300} {Suffrage:Voter ID:b03744c4-165e-b7d5-69b3-b8cd01081f9f Address:10.121.16.27:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 74
runtime:
arch = amd64
cpu_count = 2
goroutines = 131
max_procs = 2
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 69
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 56
members = 6
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 11
members = 3
query_queue = 0
query_time = 1
```
### Operating system and Environment details
Kubernetes flavour: AKS
Kubernetes version: 1.22.11
Deployment method: Helm
Chart version: 0.47.1
Consul version: 1.13.1
### Log Fragments
Error on the server:
```
2022-09-20T20:26:06.246Z [ERROR] agent.http: Request error: method=GET url=/v1/acl/token/self?stale= from=10.121.16.4:58469 error="ACL not found"
```
Error on the client:
```
consul ==> Starting Consul agent...
consul Version: '1.13.1'
consul Build Date: '2022-08-11 19:07:00 +0000 UTC'
consul Node ID: '8757ca4c-ad91-524b-a94c-6332636353ee'
consul Node name: 'aks-agentpool1-22038821-vmss000001'
consul Datacenter: 'aks01' (Segment: '')
consul Server: false (Bootstrap: false)
consul Client Addr: [0.0.0.0] (HTTP: -1, HTTPS: 8501, gRPC: 8502, DNS: 8600)
consul Cluster Addr: 10.121.12.50 (LAN: 8301, WAN: 8302)
consul Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: true
consul ==> Log data will now stream in as it occurs:
consul 2022-09-20T19:29:27.752Z [WARN] agent: The 'ca_file' field is deprecated. Use the 'tls.defaults.ca_file' field instead.
consul 2022-09-20T19:29:27.752Z [WARN] agent: The 'verify_outgoing' field is deprecated. Use the 'tls.defaults.verify_outgoing' field instead.
consul 2022-09-20T19:29:27.850Z [WARN] agent.auto_config: The 'ca_file' field is deprecated. Use the 'tls.defaults.ca_file' field instead.
consul 2022-09-20T19:29:27.850Z [WARN] agent.auto_config: The 'verify_outgoing' field is deprecated. Use the 'tls.defaults.verify_outgoing' field instead.
consul 2022-09-20T19:29:28.149Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:29:28.256Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:29:29.335Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:29:31.647Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:29:35.739Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:29:43.914Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:30:02.618Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:30:35.688Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:31:48.965Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:34:01.058Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:39:07.372Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:47:39.756Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul 2022-09-20T19:59:29.681Z [ERROR] agent.auto_config: no auto-encrypt server addresses available for use
consul stream closed
```
Contributor guide
Research direction
Start with agent/auto-config/auto_encrypt.go at the referenced lines, then inspect the Helm-generated Consul agent command and its -retry-join configuration. Compare the addresses consumed by auto-encrypt with those supplied by the AKS deployment, and verify the expected behavior in the documented single-DC multi-Kubernetes setup. Done means the cause of the missing server addresses is established and the client can complete auto-encryption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, helm, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100