moby / moby/swarmkit

Service is not DNS resolvable from another one if containers run on different nodes

Open
#1,429 46 comments 29 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/networking kind/bug
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

I have two services running a single container each, on different nodes, using same "overlay" network. When I try to ping one container from inside the other via service name, it fails:

ping akka-test
ping: bad address 'akka-test'

After I scaled the akka-test service so that a container runs on the node where the other container is running, everything suddenly starts to work.

So my questing is: is my assumption valid that services should be discoverable across entire Swarm? I mean, name of a service should be DNS resolvable from any other container in this Swarm, no matter where containers are running.

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
255fedab2fc4        bridge              bridge              local
9a450f033c48        docker_gwbridge     bridge              local
6e76844033f8        host                host                local
dzwgdein8cxa        ingress             overlay             swarm
54uqc60vx1i5        net2                overlay             swarm
d632a42ef140        none                null                local
$ docker service ls
ID            NAME         REPLICAS  IMAGE                             COMMAND
0wyv4gq14mnu  akka-test    8/8       xxxx:5000/akkahttp1:1.20
cg7r4ius7xfm  akka-test-2  1/1       xxxx:5000/akkahttp1:1.20
$ docker service inspect --pretty akka-test
ID:             0wyv4gq14mnuj8kfolizh1h23
Name:           akka-test
Mode:           Replicated
 Replicas:      8
Placement:
UpdateConfig:
 Parallelism:   1
 On failure:    pause
ContainerSpec:
 Image:         xxxx:5000/akkahttp1:1.20
Resources:
Networks: 54uqc60vx1i57d3qnmhza82c4
$ docker service inspect --pretty akka-test-2
ID:             cg7r4ius7xfmgvazmptvarn2k
Name:           akka-test-2
Mode:           Replicated
 Replicas:      1
Placement:
UpdateConfig:
 Parallelism:   1
 On failure:    pause
ContainerSpec:
 Image:         xxxx:5000/akkahttp1:1.20
Resources:
Networks: 54uqc60vx1i57d3qnmhza82c4
$ docker info
Containers: 75
 Running: 11
 Paused: 0
 Stopped: 64
Images: 42
Server Version: 1.12.1-rc1
Storage Driver: devicemapper
 Pool Name: docker-253:0-135409124-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 8.291 GB
 Data Space Total: 107.4 GB
 Data Space Available: 40.86 GB
 Metadata Space Used: 19.61 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.128 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: active
 NodeID: ao1wz862t6n4yog4hpi4yqm20
 Is Manager: true
 ClusterID: 3hpbbe2jtdoqe1zvxs41cycoq
 Managers: 3
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: xxxx
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 56
Total Memory: 188.6 GiB
Name: xxxx
ID: OWEH:OIIR:7NZ6:IKZV:RFJ4:NXAZ:NH7H:WPLC:D457:DKGN:CH2C:E2UE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with two services on the net2 overlay network, placing their containers on different nodes as described. Check whether service-name DNS resolution works across nodes in the reported Docker 1.12.1-rc1 Swarm setup; done means confirming the expected behavior or documenting the limitation and its conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.