etcd-io / etcd-io/etcd

healthcheck breaks the swarm network.

Open
#22,400 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 3h
Merged PRs (30d)
42

Description

### Bug report criteria

- [x] This bug report is not security related, security issues should be disclosed privately via [the report form](https://github.com/etcd-io/etcd/security/advisories/new).
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

Hi.

I have a swarm cluster with 3 manager nodes (manager-01,manager-02,manager-03).
I deploy a global placement service etcd.

```
services:

etcd:
hostname: etcd-{{.Node.Hostname}}
image: quay.io/coreos/etcd:v3.7.1
networks:
- etcd
environment:
ETCD_NAME: "etcd-{{.Node.Hostname}}"
ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS: "http://etcd-{{.Node.Hostname}}:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd-{{.Node.Hostname}}:2380"
ETCD_INITIAL_CLUSTER: ${ETCD_INITIAL_CLUSTER:?error}
ETCD_INITIAL_CLUSTER_STATE: ${ETCD_INITIAL_CLUSTER_STATE:-existing}
volumes:
- etcd-data:/var/lib/etcd
deploy:
mode: global
placement:
constraints:
- node.role == manager
restart_policy:
condition: any
```

and it run smoothly...
raft votes etc... I am a happy person.

if i try to put a healthcheck
```
healthcheck:
test: ["CMD", "curl", "http://127.0.0.1:2379/health/serializable=true"]
interval: 5s
timeout: 5s
retries: 6
start_period: 5s
```
Then the etcd-services tasks can not see each other anymore.and the raft vote can not be done.
I deploy a another test service with a alpine image , at the same network.

When i deploy without healthcheck everything smooth again.
- Task of the etcd-services can see each-other
- the test-service task can see all of thems.

When i put the healthcheck at the etcd-service ,
- the etcd-service services can not see each other anymore.and the raft vote can not be done.
- the test service task can't see them.

### What did you expect to happen?

1. run smoothly with the healthcheck also.

### How can we reproduce it (as minimally and precisely as possible)?

1. Create a swarm cluster with 3 manager nodes named (manager-01,manager-02,manager-03).
2. Deploy the fellow docker compose file with the .env.example file.
```
services:

etcd:
hostname: etcd-{{.Node.Hostname}}
image: quay.io/coreos/etcd:v3.7.1
networks:
- etcd
environment:
ETCD_NAME: "etcd-{{.Node.Hostname}}"
ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS: "http://etcd-{{.Node.Hostname}}:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd-{{.Node.Hostname}}:2380"
ETCD_INITIAL_CLUSTER: ${ETCD_INITIAL_CLUSTER:?error}
ETCD_INITIAL_CLUSTER_STATE: ${ETCD_INITIAL_CLUSTER_STATE:-existing}
volumes:
- etcd-data:/var/lib/etcd
deploy:
mode: global
placement:
constraints:
- node.role == manager
restart_policy:
condition: any
```

if it run smoothly...raft votes etc... and you are a Happy person also

3. add the healthckeck.
```
healthcheck:
test: ["CMD", "curl", "http://127.0.0.1:2379/health/serializable=true"]
interval: 5s
timeout: 5s
retries: 6
start_period: 5s
```
(modify the .env file ETCD_INITIAL_CLUSTER_STATE=existing)

4. deploy again.

### Anything else we need to know?

.env.example

`
ETCD_INITIAL_CLUSTER="etcd-manager-01=http://etcd-manager-01:2380,etcd-manager-02=http://etcd-manager-02:2380,etcd-manager-03=http://etcd-manager-03:2380"
ETCD_INITIAL_CLUSTER_STATE=new
`

my swarm network
`
etcd:
driver: overlay
attachable: false
internal: true
driver_opts:
encrypted: "true"
ipam:
config:
- subnet: 10.10.2.0/24
`

### Etcd version (please run commands below)

v3.7.1

### Etcd configuration (command line flags or environment variables)

# paste your configuration here

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints= endpoint status -w table
# paste output here
```

### Relevant log output

```Shell

```

Contributor guide

Open the contributing guide

Research direction

Start with the provided Docker Compose service definition and .env.example, reproducing the three-manager swarm deployment with and without the healthcheck. Compare task connectivity and raft behavior, then run `etcdctl member list -w table` and the endpoint status command from the report. Done means the three etcd tasks remain connected and raft voting succeeds with the healthcheck enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
devops, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.