hashicorp / hashicorp/consul

Agents can't find new server(s)

Open
#9,235 3 comments 0 reactions 2 assignees Claimed by @zalimeni View on GitHub
theme/kubernetes type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I have a very similar issue to #6672 but the proposed solution doesn't work for me.

I run a single-node consul server cluster inside k8s. If the pod is recreated the clients outside of k8s show weird errors and cannot reconnect. If I restart the agents using `systemctl restart consul` they connect fine.

The solution proposed in #6672 is to add
```
"skip_leave_on_interrupt": true,
"leave_on_terminate" : false,
```
to the config. As far as I understand this is the default for the server anyway. Adding it to the client didn't help.

I do understand that the whole server should not go down, but even in a k8s setup this _could_ happen. Why can the agents not handle this? All that would need to happen is an `exit 1` since systemd whould start it again anyway. How can I achieve that? Or what's the proper solution?

Btw. a 3-node consul server setup inside k8s didn't work for me due to #7750 but as mentioned, also there a complete server down could happen.

FWIW: I'm only using consul for its DNS capabilities.

#### Reproduction Steps

1. Create a cluster with (at least?) 1 client nodes and 1 server node. Server inside kubernetes, using the helm chart and host network. Client outside.
1. Destroy the server pod. K8s will recreate it.
1. Client will log the errors and not join the new cluster. If I restart the client it connects fine.

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 12b16df3
version = 1.8.4
consul:
acl = disabled
known_servers = 1
server = false
runtime:
arch = amd64
cpu_count = 8
goroutines = 47
max_procs = 8
os = linux
version = go1.14.6
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 24296
members = 6
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease = beta3
revision = a00f8cce
version = 1.9.0
consul:
acl = disabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.110.0.11:8300
server = true
raft:
applied_index = 79921
commit_index = 79921
fsm_pending = 0
last_contact = 0
last_log_index = 79921
last_log_term = 7
last_snapshot_index = 65591
last_snapshot_term = 7
latest_configuration = [{Suffrage:Voter ID:170ef5fb-0aa0-1f1c-64ad-1f73fe020bcc Address:10.110.0.11:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 7
runtime:
arch = amd64
cpu_count = 8
goroutines = 118
max_procs = 8
os = linux
version = go1.14.11
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 24335
members = 6
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6
members = 1
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Server runs inside a kubernetes cluster in digitalocean on host network. It was installed using the helm chart v. 0.25.0 with [this PR](https://github.com/hashicorp/consul-helm/pull/475) applied. I'm using the consul beta release but the same error appeard with the stable release. The agent is configured using `retry_join` with the cloud auto-joining configured.

### Log Fragments

client repeatedly logs this (log level `trace`) when the server pod is restarted:

```
2020-11-19T11:31:41.536Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2020-11-19T11:31:55.323Z [ERROR] agent: Coordinate update error: error="No known Consul servers"
2020-11-19T11:32:04.413Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2020-11-19T11:32:11.170Z [ERROR] agent: Coordinate update error: error="No known Consul servers"
2020-11-19T11:32:29.474Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2020-11-19T11:32:33.289Z [ERROR] agent: Coordinate update error: error="No known Consul servers"
2020-11-19T11:32:50.384Z [ERROR] agent: Coordinate update error: error="No known Consul servers"
2020-11-19T11:32:55.866Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2020-11-19T11:33:08.380Z [ERROR] agent: Coordinate update error: error="No known Consul servers"
```

Actually when I tried yesterday (with log level `debug`) it repeatedly printed the following...:
```
2020-11-18T13:43:40.455Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2020-11-18T13:43:45.278Z [ERROR] agent.dns: rpc error: error="No known Consul servers"
2020-11-18T13:43:45.278Z [ERROR] agent.dns: rpc error: error="No known Consul servers"
2020-11-18T13:43:45.278Z [ERROR] agent.dns: rpc error: error="No known Consul servers"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.