moby / moby/libnetwork

Docker(swarm) container failing to cummincate for some port in different subnet,but works in same subnet

Open
#2,340 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

While deploying docker etcd container on nodes on the different subnet in different datacentre.getting the below-mentioned error.

2019-02-21 16:52:42.506314 I | raft: b8b747c74aaea686 is starting a new election at term 928
2019-02-21 16:52:42.506344 I | raft: b8b747c74aaea686 became candidate at term 929
2019-02-21 16:52:42.506353 I | raft: b8b747c74aaea686 received MsgVoteResp from b8b747c74aaea686 at term 929
2019-02-21 16:52:42.506361 I | raft: b8b747c74aaea686 [logterm: 1, index: 3] sent MsgVote request to b3504381e8ba3cb at term 929
2019-02-21 16:52:42.506367 I | raft: b8b747c74aaea686 [logterm: 1, index: 3] sent MsgVote request to f572fdfc5cb68406 at term 929
2019-02-21 16:52:43.158372 W | rafthttp: health check for peer b3504381e8ba3cb could not connect: dial tcp 10.0.2.81:2380: i/o timeout
2019-02-21 16:52:43.159658 W | rafthttp: health check for peer f572fdfc5cb68406 could not connect: dial tcp 10.0.2.83:2380: i/o timeout


docker version:
[user-docker@f1cloud2201 ~]$ docker version
Client:
Version: 17.12.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:10:14 2017
OS/Arch: linux/amd64

Server:
Engine:
Version: 17.12.0-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:12:46 2017
OS/Arch: linux/amd64
Experimental: true

Followed all the pre-requieste of docker swarm port constraints.
telnet and nc also working


etcdc compose file:

version: '3'
services:
etcd01:
image: quay.io/coreos/etcd
ports:
- 2379
- 4001
- 2380
networks:
- dbs1
volumes:
- etcd01:/etcd_data
deploy:
placement:
constraints:
- node.labels.type20 == etcd01
replicas: 1

command:
  - /usr/local/bin/etcd
  - -name
  - etcd01
  - --data-dir
  - /etcd_data
  - -advertise-client-urls
  - http://etcd01:2379,http://etcd01:4001,http://127.0.0.1:2379
  - -listen-client-urls
  - http://0.0.0.0:2379,http://0.0.0.0:4001
  - -initial-advertise-peer-urls
  - http://etcd01:2380
  - -listen-peer-urls
  - http://0.0.0.0:2380
  - -initial-cluster
  - etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380

etcd02:
image: quay.io/coreos/etcd
ports:
- 2379
- 4001
- 2380
volumes:
- etcd02:/etcd_data
networks:
- dbs1
deploy:
placement:
constraints:
- node.labels.type20 == etcd02
replicas: 1
command:
- /usr/local/bin/etcd
- -name
- etcd02
- --data-dir
- /etcd_data
- -advertise-client-urls
- http://etcd02:2379,http://etcd02:4001,http://127.0.0.1:2379
- -listen-client-urls
- http://0.0.0.0:2379,http://0.0.0.0:4001
- -initial-advertise-peer-urls
- http://etcd02:2380
- -listen-peer-urls
- http://0.0.0.0:2380
- -initial-cluster
- etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380
etcd03:
image: quay.io/coreos/etcd
ports:
- 2379
- 4001
- 2380
volumes:
- etcd03:/etcd_data
networks:
- dbs1
deploy:
placement:
constraints:
- node.labels.type20 == etcd03
replicas: 1

command:
  - /usr/local/bin/etcd
  - -name
  - etcd03
  - --data-dir
  - /etcd_data
  - -advertise-client-urls
  - http://etcd03:2379,http://etcd03:4001,http://127.0.0.1:2379
  - -listen-client-urls
  - http://0.0.0.0:2379,http://0.0.0.0:4001
  - -initial-advertise-peer-urls
  - http://etcd03:2380
  - -listen-peer-urls
  - http://0.0.0.0:2380
  - -initial-cluster
  - etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380

volumes:
etcd01:
etcd02:
etcd03:

networks:
dbs1:
external: true

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided etcd Compose configuration and the Docker Swarm overlay network, focusing on peer traffic to port 2380 between nodes in different subnets. Reproduce the timeout and compare it with the stated telnet and nc results; done should mean identifying a confirmed networking cause and documenting a reproducible resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.