Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured

#### Overview of the Issue

Hello!

I have deployed cluster of Consul servers with 3 instances. Seems everything is fine unless I restart one of the nodes. After that I get `agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured` every three seconds. That's at least annoying, at most flooding the whole logs with that spam. The restarted node joins the cluster and operates normally.

Is it bug? Or do I have some real misconfiguration?

---

#### Reproduction Steps

1. Create 3 VMs, ensure all of them are in the same network and may be pinged from each other.
2. Install docker/podman. (Actually I use podman now, but I have had the same problem in docker)
3. On each VM deploy (`podman-compose up -d`) the Consul server with compose.yml and config.hcl file listed below.
4. Ensure all three Consul services started and joined the cluster.
5. Restart any of these nodes (`podman-compose restart`)
6. GET A TONS OF SPAM IN THE LOGS with `agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured`

### Consul info for the Server

compose.yml

```yaml
services:
consul:
image: docker.io/hashicorp/consul:1.17
command: agent -advertise= -node=consul-
restart: unless-stopped
volumes:
- ./consul/config:/consul/config
- ./consul/data:/consul/data
ports:
- '8300:8300'
- '8301:8301/tcp'
- '8301:8301/udp'
- '8302:8302/tcp'
- '8302:8302/udp'
- '8500:8500'
- '8600:8600/tcp'
- '8600:8600/udp'
```

consul info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 9bcafa24
version = 1.17.1
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 192.168.200.30:8300
server = true
raft:
applied_index = 333
commit_index = 333
fsm_pending = 0
last_contact = 30.612209ms
last_log_index = 333
last_log_term = 3
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:cfbde6ee-e125-057c-e9b4-08b5c48799ba Address:192.168.200.30:8300} {Suffrage:Voter ID:58b845ff-68cb-13a6-64df-e7787c690c72 Address:192.168.200.10:8300} {Suffrage:Voter ID:bb253edf-6eb9-dfb6-51b2-210e638a3fc2 Address:192.168.200.20:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 3
runtime:
arch = amd64
cpu_count = 1
goroutines = 159
max_procs = 1
os = linux
version = go1.20.12
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 3
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6
members = 3
query_queue = 0
query_time = 1
```

Server agent HCL config

```hcl
# General parameters
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
datacenter = "my-dc"
data_dir = "/consul/data"
protocol = 3
server = true

# Bootstrap Parameters
bootstrap_expect = 3

# DNS and Domain Parameters
domain = "my-project.local."

# Join Parameters
#rejoin_after_leave = true
retry_join = [ "192.168.200.10", "192.168.200.20", "192.168.200.30" ]

# Log Parameters
log_level = "INFO"

# Node Parameters
disable_host_node_id = true

# Raft Parameters
raft_protocol = 3

# UI Parameters
ui_config {
enabled = true
}
```

### Operating system and Environment details

uname -a

`Linux my-cluster-1 6.1.0-11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08) x86_64 GNU/Linux`

podman info

```
host:
arch: amd64
buildahVersion: 1.28.2
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon_2.1.6+ds1-1_amd64
path: /usr/bin/conmon
version: 'conmon version 2.1.6, commit: unknown'
cpuUtilization:
idlePercent: 98.47
systemPercent: 0.65
userPercent: 0.88
cpus: 1
distribution:
codename: bookworm
distribution: debian
version: "12"
eventLogger: journald
hostname: postgres-cluster-1
idMappings:
gidmap: null
uidmap: null
kernel: 6.1.0-11-amd64
linkmode: dynamic
logDriver: journald
memFree: 155844608
memTotal: 986537984
networkBackend: netavark
ociRuntime:
name: crun
package: crun_1.8.1-1+deb12u1_amd64
path: /usr/bin/crun
version: |-
crun version 1.8.1
commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
rundir: /run/user/0/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
exists: true
path: /run/podman/podman.sock
security:
apparmorEnabled: true
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns_1.2.0-1_amd64
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 1023406080
swapTotal: 1023406080
uptime: 24h 56m 54.00s (Approximately 1.00 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /usr/share/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 1
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphRootAllocated: 40418947072
graphRootUsed: 3992920064
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 6
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 4.3.1
Built: 0
BuiltTime: Wed Dec 31 18:00:00 1969
GitCommit: ""
GoVersion: go1.19.8
Os: linux
OsArch: linux/amd64
Version: 4.3.1
```

### Log Fragments

On *consul-1* node:

```
2023-12-15T12:49:03.243Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-3 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:49:04.245Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-2 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:49:05.245Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-3 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:49:06.246Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-2 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:49:07.251Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to consul-2.my-dc over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:49:08.243Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-2 over TCP but UDP probes failed, network may be misconfigured
```

On *consul-2* node:

```
2023-12-15T12:24:21.654Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:24:23.654Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:24:25.652Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to consul-1.my-dc over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:24:27.655Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:24:29.656Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:24:31.654Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:24:32.656Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
```

On *consul-3* node:

```
2023-12-15T12:49:57.769Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:49:59.769Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:50:00.771Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:50:03.769Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:50:06.768Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:50:08.769Z [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to consul-1 over TCP but UDP probes failed, network may be misconfigured
2023-12-15T12:50:09.768Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to consul-1.my-dc over TCP but UDP probes failed, network may be misconfigured
```