hashicorp / hashicorp/consul

Reproducible grpcWeirdError

Open
#19,848 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

## Overview of the Issue

When upgrading Consul with the [Helm chart](https://github.com/hashicorp/consul-k8s/tree/main/charts/consul) and the `updatePartition` has been decremented to `0` (i.e. all pods have been upgraded), setting the cluster leader back to `consul-server-0` fails with the text from [grpcWeirdError](https://github.com/hashicorp/consul/blob/main/testing/deployer/sprawl/peering.go#L24). Leadership is able to transfer back and forth between `consul-server-{1,2}`. Terminating the follower of `consul-server-{1,2}` allows leadership to be transferred back to `consul-server-0`. I have performed several upgrades and at each stage of each upgrade, the pod with most recent/youngest pod is unable to become the leader.

## Reproduction Steps
Steps to reproduce this issue, eg:

1. Create a cluster with 3 server nodes
1. increment `global.imageTag` to a new version
1. `helm upgrade`
1. Decrement `server.updatePartition`
1. `helm upgrade`
1. Confirm that `consul-server-2` has been upgraded and is thus the youngest node
1. Execute `consul operator raft transfer-leader -id=""`
1. See `Success` message
1. Run `consul operator raft list-peers`
1. See that `consul-server-2` is _not_ the leader

### Consul info for Server
Servers have exhibited this behavior for Consul 1.16.1, 1.16.2, and 1.16.3. Clients are running the same container image, but have not exhibited any noteworthy behavior.
Server info:
```
/consul/config $ consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 68f81912
version = 1.16.2
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.2.47.227:8300
server = true
raft:
applied_index = 5990626005
commit_index = 5990626005
fsm_pending = 0
last_contact = 22.280224ms
last_log_index = 5990626005
last_log_term = 350
last_snapshot_index = 5990623773
last_snapshot_term = 320
latest_configuration = [{Suffrage:Voter ID:644ec03b-bee9-924e-28e5-29180b1238ec Address:10.2.31.129:8300} {Suffrage:Voter ID:226a49e3-2662-a59d-132f-425fba295114 Address:10.2.47.227:8300} {Suffrage:Voter ID:660f76ad-ddc5-0e48-a4f3-b7d88b38fe60 Address:10.2.8.161:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 350
runtime:
arch = amd64
cpu_count = 4
goroutines = 188
max_procs = 4
os = linux
version = go1.20.8
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 81
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 17779
members = 9
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 195
members = 3
query_queue = 0
query_time = 1
```

### Operating system and Environment details
We're running on AWS EKS, kubernetes server version v1.28.3-eks-4f4795d.

### Log Fragments
```
consul-server-0 consul 2023-12-06T18:17:44.484Z [WARN] agent: [core][Channel #19 SubChannel #891] grpc: addrConn.createTransport failed to connect to {
consul-server-0 consul "Addr": "dc1-",
consul-server-0 consul "ServerName": "leader",
consul-server-0 consul "Attributes": null,
consul-server-0 consul "BalancerAttributes": null,
consul-server-0 consul "Type": 0,
consul-server-0 consul "Metadata": null
consul-server-0 consul }. Err: connection error: desc = "transport: Error while dialing: failed to find Consul server for global address \"dc1-\""
```

Contributor guide

Open the contributing guide

Research direction

Start with the Helm chart upgrade sequence and the grpcWeirdError reference in testing/deployer/sprawl/peering.go. Reproduce the three-server upgrade with updatePartition set to 0, then run raft transfer-leader and raft list-peers while reviewing the supplied Consul logs. Done means the upgraded youngest server can become leader and list-peers reports it as leader.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, helm, kubernetes
Domain
cloud, devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.