Docker fails to provide DNS/name/alias to containers in a overlay network when is created with IPv6

Description

TL; DR jump to Reproduce Steps / Expected behavior

In the attempt to upgrade de deploy scenario to enable some services open IPv6 socket for some itchy IoT context, we are trying to ship IPv6 ULA inside the container to then be able to do so.

Some containers the DNS fails and fall to localhost, others the resolution fall to the IPv4. At least going with IPv4 is OK because the services is able to send package with each other using their network alias/container name:

# ping -c 1 ingress_certman
PING ingress_certman (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.054 ms

--- ingress_certman ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.054/0.054/0.054/0.000 ms
# ping -c 1 ingress_haproxy
PING ingress_haproxy (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.076 ms

--- ingress_haproxy ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.076/0.076/0.076/0.000 ms
# ping -c 1 service_a
PING service_a (172.22.32.7) 56(84) bytes of data.
64 bytes from service_a.1.x7jjrqpal8rjgphp99us4d32j.swarm_overlay_network (172.22.32.7): icmp_seq=1 ttl=64 time=0.518 ms

--- service_a ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.518/0.518/0.518/0.000 ms
# ping -c 1 service_b
PING service_b (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.036 ms

--- service_b ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.036/0.036/0.036/0.000 ms
# ping -c 1 service_c
PING service_c (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.035 ms

--- service_c ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.035/0.035/0.035/0.000 ms
# ping -c 1 service_d
PING service_d (172.22.32.3) 56(84) bytes of data.
64 bytes from service_d.1.x7w6x3y65cs1clun51s7ufvxf.swarm_overlay_network (172.22.32.3): icmp_seq=1 ttl=64 time=0.232 ms

--- service_d ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.232/0.232/0.232/0.000 ms
# ping -c 1 service_e
PING service_e (172.22.32.2) 56(84) bytes of data.
64 bytes from service_e.1.nnifldzdv8heib4genmrvhkgc.swarm_overlay_network (172.22.32.2): icmp_seq=1 ttl=64 time=0.734 ms

--- service_e ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.734/0.734/0.734/0.000 ms
# ping -c 1 service_f
PING service_f (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.048 ms

--- service_f ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.048/0.048/0.048/0.000 ms

• Can direclty ping IPv4 of containers:

# ping -c 1 172.22.32.9
PING 172.22.32.9 (172.22.32.9) 56(84) bytes of data.
64 bytes from 172.22.32.9: icmp_seq=1 ttl=64 time=0.094 ms

--- 172.22.32.9 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.094/0.094/0.094/0.000 ms
# ping -c 1 172.22.32.10
PING 172.22.32.10 (172.22.32.10) 56(84) bytes of data.
64 bytes from 172.22.32.10: icmp_seq=1 ttl=64 time=0.216 ms

--- 172.22.32.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.216/0.216/0.216/0.000 ms
# ping -c 1 172.22.32.7
PING 172.22.32.7 (172.22.32.7) 56(84) bytes of data.
64 bytes from 172.22.32.7: icmp_seq=1 ttl=64 time=0.150 ms

--- 172.22.32.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.150/0.150/0.150/0.000 ms
# ping -c 1 172.22.32.5
PING 172.22.32.5 (172.22.32.5) 56(84) bytes of data.
64 bytes from 172.22.32.5: icmp_seq=1 ttl=64 time=0.190 ms

--- 172.22.32.5 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.190/0.190/0.190/0.000 ms
# ping -c 1 172.22.32.8
PING 172.22.32.8 (172.22.32.8) 56(84) bytes of data.
64 bytes from 172.22.32.8: icmp_seq=1 ttl=64 time=0.212 ms

--- 172.22.32.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.212/0.212/0.212/0.000 ms
# ping -c 1 172.22.32.3
PING 172.22.32.3 (172.22.32.3) 56(84) bytes of data.
64 bytes from 172.22.32.3: icmp_seq=1 ttl=64 time=0.113 ms

--- 172.22.32.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.113/0.113/0.113/0.000 ms
# ping -c 1 172.22.32.2
PING 172.22.32.2 (172.22.32.2) 56(84) bytes of data.
64 bytes from 172.22.32.2: icmp_seq=1 ttl=64 time=0.092 ms

--- 172.22.32.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.092/0.092/0.092/0.000 ms
# ping -c 1 172.22.32.6
PING 172.22.32.6 (172.22.32.6) 56(84) bytes of data.
64 bytes from 172.22.32.6: icmp_seq=1 ttl=64 time=0.212 ms

--- 172.22.32.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.212/0.212/0.212/0.000 ms

• Can directly ping ULA IPv6 of containers:

# ping -c 1 fd46:f722:3989::13
PING fd46:f722:3989::13(fd46:f722:3989::13) 56 data bytes
64 bytes from fd46:f722:3989::13: icmp_seq=1 ttl=64 time=0.098 ms

--- fd46:f722:3989::13 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.098/0.098/0.098/0.000 ms
# ping -c 1 fd46:f722:3989::11
PING fd46:f722:3989::11(fd46:f722:3989::11) 56 data bytes
64 bytes from fd46:f722:3989::11: icmp_seq=1 ttl=64 time=0.263 ms

--- fd46:f722:3989::11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.263/0.263/0.263/0.000 ms
# ping -c 1 fd46:f722:3989::f
PING fd46:f722:3989::f(fd46:f722:3989::f) 56 data bytes
64 bytes from fd46:f722:3989::f: icmp_seq=1 ttl=64 time=0.239 ms

--- fd46:f722:3989::f ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.239/0.239/0.239/0.000 ms
# ping -c 1 fd46:f722:3989::3
PING fd46:f722:3989::3(fd46:f722:3989::3) 56 data bytes
64 bytes from fd46:f722:3989::3: icmp_seq=1 ttl=64 time=0.333 ms

--- fd46:f722:3989::3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.333/0.333/0.333/0.000 ms
# ping -c 1 fd46:f722:3989::d
PING fd46:f722:3989::d(fd46:f722:3989::d) 56 data bytes
64 bytes from fd46:f722:3989::d: icmp_seq=1 ttl=64 time=0.313 ms

--- fd46:f722:3989::d ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.313/0.313/0.313/0.000 ms
# ping -c 1 fd46:f722:3989::c
PING fd46:f722:3989::c(fd46:f722:3989::c) 56 data bytes
64 bytes from fd46:f722:3989::c: icmp_seq=1 ttl=64 time=0.236 ms

--- fd46:f722:3989::c ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.236/0.236/0.236/0.000 ms
# ping -c 1 fd46:f722:3989::b
PING fd46:f722:3989::b(fd46:f722:3989::b) 56 data bytes
64 bytes from fd46:f722:3989::b: icmp_seq=1 ttl=64 time=0.306 ms

--- fd46:f722:3989::b ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.306/0.306/0.306/0.000 ms
# ping -c 1 fd46:f722:3989::a
PING fd46:f722:3989::a(fd46:f722:3989::a) 56 data bytes
64 bytes from fd46:f722:3989::a: icmp_seq=1 ttl=64 time=0.333 ms

--- fd46:f722:3989::a ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.333/0.333/0.333/0.000 ms

• Extra: No resolution for ipv6:

  # ping -6 service_a
  ping: service_a: Address family for hostname not supported

Reproduce

• Stop docker in the system if already running:

sudo systemctl stop docker

• Edit the /etc/docker/daemon.json file to enable ipv6 in docker:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd54:69d7:8d50::/48",
  "experimental": true,
  "ip6tables": true
}

• Restart docker:

sudo systemctl restart docker

• Create gwbridge before swarm init with ipv6 subnet:

sudo docker network create \
  --ipv6 \
  --subnet 172.20.0.0/20 \
  --gateway 172.20.0.1 \
  --gateway fd0e:448e:22a5::1 \
  --subnet fd0e:448e:22a5::/48 \
  --opt com.docker.network.bridge.name=docker_gwbridge \
  --opt com.docker.network.bridge.enable_icc=false \
  --opt com.docker.network.bridge.enable_ip_masquerade=true \
  docker_gwbridge

• Then init the swarm:

sudo docker swarm init

• Create the swarm_overlay_network to be used by the stacks containers:

sudo docker network create --ipv6 --subnet fd46:f722:3989::/48 --subnet=172.22.32.0/24 --scope=swarm -d overlay swarm_overlay_network

• Get the docker-compose made to reproduce the problem and deploy as a stack in the swarm:

git clone https://github.com/gabrielxfs/ping-pong.git
cd ping-pong
sudo docker stack deploy -c ./docker-compose.yml pp

Expected behavior

• Inside the container, is expected that they use their internal docker DNS name and be able to send package to each other. But that is not what happens:

$ sudo docker ps
CONTAINER ID   IMAGE                    COMMAND     CREATED         STATUS         PORTS     NAMES
f41ba3783866   gabrielw1n3/ping:test    "init.sh"   6 minutes ago   Up 6 minutes   80/tcp    pp_pang.1.nx1dpfk8duz0rgtic67g8iggz
ca702d54651d   gabrielw1n3/ping:test    "init.sh"   6 minutes ago   Up 6 minutes   80/tcp    pp_pong.1.myqfmc8s6ivnefdf359v2u3ae
c24f6274de20   portainer/agent:2.19.4   "./agent"   8 minutes ago   Up 8 minutes             portainer_agent.lydbtmzoife1muovyjbh2gfhw.prajsmemzqxr8peimexb3r6r4
$ sudo docker logs f41ba3783866
Waiting for pong on 8082...
wait-for-it.sh: waiting for pong:8082 without a timeout
Example app listening on port 8081
$ sudo docker logs ca702d54651d
Waiting for pang on 8081...
wait-for-it.sh: waiting for pang:8081 without a timeout
Example app listening on port 8082
$ sudo docker exec -u root -it f41ba3783866 bash
f41ba3783866:/app# ping pong
ping: bad address 'pong'
f41ba3783866:/app# ping pp_pong
ping: bad address 'pp_pong'

• Below is the network list in portainer to depict the context:
  

• If we take a look at networks we got the IPv6 address and their containers names:
  

• We can evenly ping directly their address:

f41ba3783866:/app# ping 172.22.32.2
PING 172.22.32.2 (172.22.32.2): 56 data bytes
64 bytes from 172.22.32.2: seq=0 ttl=64 time=0.198 ms
^C
--- 172.22.32.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.198/0.198/0.198 ms
f41ba3783866:/app# ping -c fd46:f722:3989::22
ping: invalid number 'fd46:f722:3989::22'
f41ba3783866:/app# ping -c 1 fd46:f722:3989::22
PING fd46:f722:3989::22 (fd46:f722:3989::22): 56 data bytes
64 bytes from fd46:f722:3989::22: seq=0 ttl=64 time=0.207 ms

--- fd46:f722:3989::22 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.207/0.207/0.207 ms

• But, when we create the swarm_overlay_network without IPv6:
  

• The containers are able to switch packages with each other:

$ sudo docker ps
CONTAINER ID   IMAGE                    COMMAND     CREATED              STATUS              PORTS     NAMES
639a82d52273   gabrielw1n3/ping:test    "init.sh"   About a minute ago   Up About a minute   80/tcp    pp_pong.1.47l3w6niv4oihtoe9gtl7l5yp
782b344cfaa6   gabrielw1n3/ping:test    "init.sh"   About a minute ago   Up About a minute   80/tcp    pp_pang.1.jx33uxy8y0rad76hjh54amoxm
c24f6274de20   portainer/agent:2.19.4   "./agent"   23 minutes ago       Up 23 minutes                 portainer_agent.lydbtmzoife1muovyjbh2gfhw.prajsmemzqxr8peimexb3r6r4
$ sudo docker exec -u root -it 782b344cfaa6 bash
782b344cfaa6:/app# ping -c 1 pong
PING pong (172.22.32.5): 56 data bytes
64 bytes from 172.22.32.5: seq=0 ttl=64 time=0.105 ms

--- pong ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.105/0.105/0.105 ms
782b344cfaa6:/app# ping -c 1 pp_pong
PING pp_pong (172.22.32.5): 56 data bytes
64 bytes from 172.22.32.5: seq=0 ttl=64 time=0.105 ms

--- pp_pong ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.105/0.105/0.105 ms

So, semmingly, the problem lies on the the missing of DNS in docker when the network has IPv6 activated.

docker version

Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:17:51 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:17:51 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 42
  Running: 3
  Paused: 0
  Stopped: 39
 Images: 30
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: lydbtmzoife1muovyjbh2gfhw
  Is Manager: true
  ClusterID: l7mc2e7s4d7eqi3yw63shbbkg
  Managers: 1
  Nodes: 1
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.1.23  Manager Addresses:
   192.168.1.23:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-101-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.36GiB
 Name: gabriel
 ID: f941e66e-d106-40a8-8456-ec61d68c3495
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: gabrielw1n3
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Thanks!

Start with /etc/docker/daemon.json and the documented docker network commands, then deploy the ping-pong reproduction from docker-compose.yml. Verify name and alias resolution between services on an IPv6-enabled swarm overlay network; done means container names resolve and traffic works without relying on literal IPv4 or IPv6 addresses.