healthcheck breaks the swarm network.
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 42
Description
### Bug report criteria
- [x] This bug report is not security related, security issues should be disclosed privately via [the report form](https://github.com/etcd-io/etcd/security/advisories/new).
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
Hi.
I have a swarm cluster with 3 manager nodes (manager-01,manager-02,manager-03).
I deploy a global placement service etcd.
```
services:
etcd:
hostname: etcd-{{.Node.Hostname}}
image: quay.io/coreos/etcd:v3.7.1
networks:
- etcd
environment:
ETCD_NAME: "etcd-{{.Node.Hostname}}"
ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS: "http://etcd-{{.Node.Hostname}}:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd-{{.Node.Hostname}}:2380"
ETCD_INITIAL_CLUSTER: ${ETCD_INITIAL_CLUSTER:?error}
ETCD_INITIAL_CLUSTER_STATE: ${ETCD_INITIAL_CLUSTER_STATE:-existing}
volumes:
- etcd-data:/var/lib/etcd
deploy:
mode: global
placement:
constraints:
- node.role == manager
restart_policy:
condition: any
```
and it run smoothly...
raft votes etc... I am a happy person.
if i try to put a healthcheck
```
healthcheck:
test: ["CMD", "curl", "http://127.0.0.1:2379/health/serializable=true"]
interval: 5s
timeout: 5s
retries: 6
start_period: 5s
```
Then the etcd-services tasks can not see each other anymore.and the raft vote can not be done.
I deploy a another test service with a alpine image , at the same network.
When i deploy without healthcheck everything smooth again.
- Task of the etcd-services can see each-other
- the test-service task can see all of thems.
When i put the healthcheck at the etcd-service ,
- the etcd-service services can not see each other anymore.and the raft vote can not be done.
- the test service task can't see them.
### What did you expect to happen?
1. run smoothly with the healthcheck also.
### How can we reproduce it (as minimally and precisely as possible)?
1. Create a swarm cluster with 3 manager nodes named (manager-01,manager-02,manager-03).
2. Deploy the fellow docker compose file with the .env.example file.
```
services:
etcd:
hostname: etcd-{{.Node.Hostname}}
image: quay.io/coreos/etcd:v3.7.1
networks:
- etcd
environment:
ETCD_NAME: "etcd-{{.Node.Hostname}}"
ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS: "http://etcd-{{.Node.Hostname}}:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd-{{.Node.Hostname}}:2380"
ETCD_INITIAL_CLUSTER: ${ETCD_INITIAL_CLUSTER:?error}
ETCD_INITIAL_CLUSTER_STATE: ${ETCD_INITIAL_CLUSTER_STATE:-existing}
volumes:
- etcd-data:/var/lib/etcd
deploy:
mode: global
placement:
constraints:
- node.role == manager
restart_policy:
condition: any
```
if it run smoothly...raft votes etc... and you are a Happy person also
3. add the healthckeck.
```
healthcheck:
test: ["CMD", "curl", "http://127.0.0.1:2379/health/serializable=true"]
interval: 5s
timeout: 5s
retries: 6
start_period: 5s
```
(modify the .env file ETCD_INITIAL_CLUSTER_STATE=existing)
4. deploy again.
### Anything else we need to know?
.env.example
`
ETCD_INITIAL_CLUSTER="etcd-manager-01=http://etcd-manager-01:2380,etcd-manager-02=http://etcd-manager-02:2380,etcd-manager-03=http://etcd-manager-03:2380"
ETCD_INITIAL_CLUSTER_STATE=new
`
my swarm network
`
etcd:
driver: overlay
attachable: false
internal: true
driver_opts:
encrypted: "true"
ipam:
config:
- subnet: 10.10.2.0/24
`
### Etcd version (please run commands below)
v3.7.1
### Etcd configuration (command line flags or environment variables)
# paste your configuration here
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
```console
$ etcdctl member list -w table
# paste output here
$ etcdctl --endpoints= endpoint status -w table
# paste output here
```
### Relevant log output
```Shell
```
Contributor guide
Research direction
Start with the provided Docker Compose service definition and .env.example, reproducing the three-manager swarm deployment with and without the healthcheck. Compare task connectivity and raft behavior, then run `etcdctl member list -w table` and the endpoint status command from the report. Done means the three etcd tasks remain connected and raft voting succeeds with the healthcheck enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose
- Domain
- devops, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100