DNS not working after docker daemon restart (live-restore)
@ctalledo is already working on this.
Since Apr 22, 2021.
- Dominant language
- Shell
- Stars
- 3.9k
- Forks
- 230
- Avg merge
- 7h 48m
- Merged PRs (30d)
- 3
Description
We're using calico with docker (libnetwork-plugin) and every hw node has a local dnsmasq to proxy and cache DNS requests. So our docker configuration points the dns of every container to the host machine. So far everything works perfectly fine.
After restarting the docker daemon using the live-restore feature it seems that at least the iptables rules in the container nat table gets messed up, so dns requests stops working.
This is the table when a sysbox container is freshly started or restarted
# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER_OUTPUT all -- anywhere 169.254.1.1
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER_OUTPUT all -- anywhere 169.254.1.1
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
DOCKER_POSTROUTING all -- anywhere 169.254.1.1
Chain DOCKER_OUTPUT (2 references)
target prot opt source destination
DNAT tcp -- anywhere 169.254.1.1 tcp dpt:domain to:127.0.0.11:46183
DNAT udp -- anywhere 169.254.1.1 udp dpt:domain to:127.0.0.11:51249
Chain DOCKER_POSTROUTING (1 references)
target prot opt source destination
SNAT tcp -- 127.0.0.11 anywhere tcp spt:46183 to:169.254.1.1:53
SNAT udp -- 127.0.0.11 anywhere udp spt:51249 to:169.254.1.1:53
After Daemon restart additional rules appear, which are the default when not using sysbox.
iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER_OUTPUT all -- anywhere 169.254.1.1
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER_OUTPUT all -- anywhere 127.0.0.11
DOCKER_OUTPUT all -- anywhere 169.254.1.1
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
DOCKER_POSTROUTING all -- anywhere 127.0.0.11
DOCKER_POSTROUTING all -- anywhere 169.254.1.1
Chain DOCKER_OUTPUT (3 references)
target prot opt source destination
DNAT tcp -- anywhere 127.0.0.11 tcp dpt:domain to:127.0.0.11:39639
DNAT udp -- anywhere 127.0.0.11 udp dpt:domain to:127.0.0.11:43355
DNAT tcp -- anywhere 169.254.1.1 tcp dpt:domain to:127.0.0.11:46183
DNAT udp -- anywhere 169.254.1.1 udp dpt:domain to:127.0.0.11:51249
Since sysbox rewrites resolv.conf to use 169.254.1.1 as dns and not 127.0.0.11 dns requests breaks.
If you need any more information I am happy to provide them.
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.