moby / moby/libnetwork

ipv6 on ingress overlay network fails with error: Invalid address : It does not belong to any of this network's subnets

Open
#2,293 1 comment 11 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

My goal is to run services in ipv4/ipv6 dual-stack but re-creating the default ingress network for swarm mode fails.

Operating System: Ubuntu 18.04.1 LTS
Server Version: 18.06.1-ce

Re-reproduce the issue:

# install docker
ip a a dev ens4 2001:db8::1/64; \
apt-get update; apt-get upgrade -y; \
apt-get install -y apt-transport-https ca-certificates \
curl gnupg2 software-properties-common; \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -; \
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"; \
apt-get update; \
apt-get install -y docker-ce;

# enable debugging
cat <<EOF > /etc/docker/daemon.json
{
    "debug": true
}
EOF
systemctl restart docker

# enable swarm mode for overlay networking
docker swarm init

# remove ingress and docker_gwbrdige
echo y | docker network rm ingress; \
docker network rm docker_gwbridge;

# re-create docker_gwbridge and ingress with ipv6 enabled
docker network create docker_gwbridge \
--ipv6 --subnet=2001:db8:18::/64 --gateway=2001:db8:18::1 \
--subnet 172.18.0.0/16 --gateway=172.18.0.1 \
--opt com.docker.network.bridge.name=docker_gwbridge \
--opt com.docker.network.bridge.enable_icc=false \
--opt com.docker.network.bridge.enable_ip_masquerade=true; \
docker network create ingress \
--ipv6 --subnet=2001:db8:255::/64 --gateway=2001:db8:255::1 \
--subnet=10.255.0.0/16 --gateway=10.255.0.1 \
--driver overlay --ingress

Creating a service fails:

docker service create --publish 80:80 --name hello_world hello-world
ybnp2pun50xk59w7p2pnrrbtv
overall progress: 0 out of 1 tasks
1/1: Invalid address 2001:db8:255::2: It does not belong to any of this network…
^COperation continuing in background.
Use `docker service ps ybnp2pun50xk59w7p2pnrrbtv` to check progress.

looking at the dockerd debug logs reveals that the ingress network cannot be created:

level=debug msg="Calling POST /v1.38/networks/create"
level=debug msg="form data: {\"Attachable\":false,\"CheckDuplicate\":true,\"ConfigFrom\":null,\"ConfigOnly\":false,\"Driver\":\"overlay\",\"EnableIPv6\":true,\"IPAM\":{\"Config\":[{\"Gateway\":\"2001:db8:255::1\",\"Subnet\":\"2001:db8:255::/64\"},{\"Gateway\":\"10.255.0.1\",\"Subnet\":\"10.255.0.0/16\"}],\"Driver\":\"default\",\"Options\":{}},\"Ingress\":true,\"Internal\":false,\"Labels\":{},\"Name\":\"ingress\",\"Options\":{},\"Scope\":\"\"}"

level=debug msg="RequestPool(GlobalDefault, 2001:db8:255::/64, , map[], false)"
level=debug msg="RequestAddress(GlobalDefault/2001:db8:255::/64, 2001:db8:255::1, map[RequestAddressType:com.docker.network.gateway com.docker.network.ipam.serial:true])"
level=debug msg="Request address PoolID:2001:db8:255::/64 App: ipam/default/data, ID: GlobalDefault/2001:db8:255::/64, DBIndex: 0x0, Bits: 18446744073709551615, Unselected: 18446744073709551614, Sequence: (0x80000000, 1)->(0x0, 576460752303423487)->end Curr:0 Serial:true PrefAddress:2001:db8:255::1 "

level=debug msg="RequestPool(GlobalDefault, 10.255.0.0/16, , map[RequestAddressType:com.docker.network.gateway com.docker.network.ipam.serial:true], false)"
level=debug msg="RequestAddress(GlobalDefault/10.255.0.0/16, 10.255.0.1, map[RequestAddressType:com.docker.network.gateway com.docker.network.ipam.serial:true])"
level=debug msg="Request address PoolID:10.255.0.0/16 App: ipam/default/data, ID: GlobalDefault/10.255.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:true PrefAddress:10.255.0.1 "

level=debug msg="RequestAddress(GlobalDefault/2001:db8:255::/64, <nil>, map[com.docker.network.ipam.serial:true])"
level=debug msg="Request address PoolID:2001:db8:255::/64 App: ipam/default/data, ID: GlobalDefault/2001:db8:255::/64, DBIndex: 0x0, Bits: 18446744073709551615, Unselected: 18446744073709551613, Sequence: (0xc0000000, 1)->(0x0, 576460752303423487)->end Curr:0 Serial:true PrefAddress:<nil> "

level=debug msg="Allocating IPv4 pools for network ingress (j4946tge57qxjjoo1koehnu4f)"
level=debug msg="RequestPool(LocalDefault, 10.255.0.0/16, , map[], false)"
level=debug msg="RequestAddress(LocalDefault/10.255.0.0/16, 10.255.0.1, map[RequestAddressType:com.docker.network.gateway])"
level=debug msg="Request address PoolID:10.255.0.0/16 App: ipam/default/data, ID: LocalDefault/10.255.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:10.255.0.1 "

level=debug msg="overlay: Received vxlan IDs: 4097,4098"

level=debug msg="/sbin/iptables, [--wait -t mangle -C OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1048832 -j MARK --set-mark 13681891]"
level=debug msg="/sbin/iptables, [--wait -t filter -C INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1048832 -j ACCEPT]"
level=debug msg="/sbin/iptables, [--wait -t filter -C INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1048832 -j DROP]"
level=debug msg="/sbin/iptables, [--wait -t mangle -C OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j MARK --set-mark 13681891]"
level=debug msg="/sbin/iptables, [--wait -t filter -C INPUT -m policy --dir in --pol ipsec -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j ACCEPT]"
level=debug msg="/sbin/iptables, [--wait -t filter -C INPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=1049088 -j DROP]"

level=debug msg="instance-6(969aca19e4c1): joined network j4946tge57qxjjoo1koehnu4f"

level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodest_conn: no such file or directory"

level=debug msg="Assigning addresses for endpoint ingress-endpoint's interface on network ingress"
level=debug msg="instance-6(969aca19e4c1): leaving network j4946tge57qxjjoo1koehnu4f"
level=debug msg="releasing IPv4 pools from network ingress (j4946tge57qxjjoo1koehnu4f)"
level=debug msg="ReleaseAddress(LocalDefault/10.255.0.0/16, 10.255.0.1)"
level=debug msg="Released address PoolID:LocalDefault/10.255.0.0/16, Address:10.255.0.1 Sequence:App: ipam/default/data, ID: LocalDefault/10.255.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0"
level=debug msg="ReleasePool(LocalDefault/10.255.0.0/16)"
level=error msg="Failed creating ingress network: Invalid address 2001:db8:255::2: It does not belong to any of this network's subnets

Address 2001:db8:255::2 is part of network 2001:db8:255::/64, however from the above logs it looks like a RequestPool(LocalDefault, ...) has only been called for IPv4. I'm not sure whether that's intended and wonder if that is the reason for error Invalid address 2001:db8:255::2: It does not belong to any of this network's subnets as there is no LocalDefault subnet for the IPv6 network.

Any clue if this should work or whether I'm doing something wrong on the configuration side?

the attached files hopefully contain all relevant details.
docker_info.txt
docker_gwbridge.json.txt
ingress.json.txt
hello_world.json.txt
ip_info.txt
logs.txt

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ingress recreation commands and the IPAM entries in logs.txt, then compare them with ingress.json.txt and docker_gwbridge.json.txt. Trace how the IPv6 ingress endpoint address is assigned during overlay creation. Done means dual-stack ingress creation no longer reports the invalid-address error, with the reported reproduction verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.