hashicorp / hashicorp/consul

The consul log output Removing LAN server and Adding LAN server frequently when server node is healthy

Open
#15,684 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

I have a cluster with 3 server nodes and 1 client node. When I periodic down and up the client node network. The leader server log output Removing LAN server and Adding LAN server frequently and fastly(removing and adding interval 40ms)。And then my consul distribute lock session return is nil, it lost the lock.

my consul version: 1.12.1

**My clusters members:**
Node Address Status Type Build Protocol DC Partition Segment
Alexs-MacBook-Pro.local 10.0.2.1:8301 alive server 1.12.1 2 tenant default
compute 10.0.2.13:8301 alive server 1.12.1 2 tenant default
local-ceph-2 10.0.2.12:8301 alive server 1.12.1 2 tenant default
compute16 10.0.2.16:8301 alive client 1.12.1 2 tenant default

**The periodic down and up client node network script:**
while true
do
ifdown ens36
sleep 20
ifup ens36
sleep 30
done

**The log:**
2022-12-06T10:25:32.140+0800 [INFO] agent.server.serf.lan: serf: EventMemberJoin: compute16 10.0.2.16
2022-12-06T10:25:32.140+0800 [INFO] agent.server: member joined, marking health alive: member=compute16 partition=default
2022-12-06T10:25:53.957+0800 [INFO] agent.server.memberlist.lan: memberlist: Suspect compute16 has failed, no acks received
2022-12-06T10:25:56.963+0800 [INFO] agent.server.memberlist.lan: memberlist: Suspect compute16 has failed, no acks received
2022-12-06T10:25:57.954+0800 [INFO] agent.server.memberlist.lan: memberlist: Marking compute16 as failed, suspect timeout reached (2 peer confirmations)
2022-12-06T10:25:57.954+0800 [INFO] agent.server.serf.lan: serf: EventMemberFailed: compute16 10.0.2.16
2022-12-06T10:25:57.954+0800 [INFO] agent.server: member failed, marking health critical: member=compute16 partition=default
2022-12-06T10:26:13.674+0800 [INFO] agent.server.serf.lan: serf: EventMemberJoin: compute16 10.0.2.16
2022-12-06T10:26:13.675+0800 [INFO] agent.server: member joined, marking health alive: member=compute16 partition=default
2022-12-06T10:26:13.675+0800 [INFO] agent.server.serf.lan: serf: EventMemberFailed: local-ceph-2 10.0.2.12
2022-12-06T10:26:13.675+0800 [INFO] agent.server: **Removing LAN server**: server="local-ceph-2 (Addr: tcp/10.0.2.12:8300) (DC: tenant)"
2022-12-06T10:26:13.707+0800 [INFO] agent.server: member failed, marking health critical: member=local-ceph-2 partition=default
2022-12-06T10:26:13.734+0800 [WARN] agent.server.raft: unable to get address for server, using fallback address: id=f59e54fc-f6b9-8b16-2b7f-0fd7102b2ca0 fallback=10.0.2.12:8300 error="Could not find address for server id f59e54fc-f6b9-8b16-2b7f-0fd7102b2ca0"
2022-12-06T10:26:13.736+0800 [WARN] agent.server.raft: unable to get address for server, using fallback address: id=f59e54fc-f6b9-8b16-2b7f-0fd7102b2ca0 fallback=10.0.2.12:8300 error="Could not find address for server id f59e54fc-f6b9-8b16-2b7f-0fd7102b2ca0"
2022-12-06T10:26:13.761+0800 [INFO] agent.server.serf.lan: serf: EventMemberJoin: local-ceph-2 10.0.2.12
2022-12-06T10:26:13.761+0800 [INFO] agent.server: **Adding LAN server**: server="local-ceph-2 (Addr: tcp/10.0.2.12:8300) (DC: tenant)"
2022-12-06T10:26:13.761+0800 [INFO] agent.server: member joined, marking health alive: member=local-ceph-2 partition=default

**My server node config:**
3 server nodes is the same,expect "bind_addr" and "addresses"
{
"bootstrap_expect": 3,
"bind_addr": "10.0.2.12",
"datacenter": "tenant",
"data_dir": "/tmp/consul_tenant",
"log_file": "/var/log/consul/tenant.log",
"log_level": "DEBUG",
"server": true,
"addresses": {
"grpc": "10.0.2.12",
"https": "10.0.2.12",
"dns": "10.0.2.12",
"http": "10.0.2.12"
},
"ports": {
"http": 8500,
"grpc": -1,
"serf_lan": 8301,
"serf_wan": -2,
"dns": -3,
"server": 8300
},
"enable_script_checks": false,
"retry_join": ["10.0.2.12"]
}

**My client node config:**
{
"bind_addr": "10.0.2.16",
"datacenter": "tenant",
"data_dir": "/tmp/consul_tenant",
"log_file": "/var/log/consul/tenant.log",
"log_level": "DEBUG",
"addresses": {
"grpc": "10.0.2.16",
"https": "10.0.2.16",
"dns": "10.0.2.16",
"http": "10.0.2.16"
},
"ports": {
"http": 8500,
"grpc": -1,
"serf_lan": 8301,
"serf_wan": -2,
"dns": -3,
"server": 8300
},
"enable_script_checks": false,
"retry_join": ["10.0.2.12"]
}

**My systemd start exec:**
[Service]
LimitNOFILE=131072
LimitNPROC=131072
Environment=LIBGUESTFS_ATTACH_METHOD=appliance
Type=simple
NotifyAccess=all
TimeoutStartSec=0
Restart=always
RestartSec=5
ExecStart=/usr/bin/consul agent -config-file /etc/consul.d/tenant.json

Contributor guide

Open the contributing guide

Research direction

Start with the provided ifdown/ifup loop, the server and client configs in /etc/consul.d/tenant.json, and the LAN membership logs. Trace the failed and joined events alongside the lock-session loss; no source file or test is named, so identify the relevant entry point before making changes. Done means explaining and correcting the rapid server membership churn without losing the lock.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.