nestybox / nestybox/sysbox

DNS not working after docker daemon restart (live-restore)

Open
#270 6 comments 0 reactions 1 assignee View on GitHub

@ctalledo is already working on this.

Since Apr 22, 2021.

bug
Dominant language
Shell
Stars
3.9k
Forks
230
Avg merge
7h 48m
Merged PRs (30d)
3

Description

We're using calico with docker (libnetwork-plugin) and every hw node has a local dnsmasq to proxy and cache DNS requests. So our docker configuration points the dns of every container to the host machine. So far everything works perfectly fine.

After restarting the docker daemon using the live-restore feature it seems that at least the iptables rules in the container nat table gets messed up, so dns requests stops working.

This is the table when a sysbox container is freshly started or restarted

# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER_OUTPUT  all  --  anywhere             169.254.1.1
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER_OUTPUT  all  --  anywhere             169.254.1.1
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER_POSTROUTING  all  --  anywhere             169.254.1.1
Chain DOCKER_OUTPUT (2 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere             169.254.1.1          tcp dpt:domain to:127.0.0.11:46183
DNAT       udp  --  anywhere             169.254.1.1          udp dpt:domain to:127.0.0.11:51249
Chain DOCKER_POSTROUTING (1 references)
target     prot opt source               destination
SNAT       tcp  --  127.0.0.11           anywhere             tcp spt:46183 to:169.254.1.1:53
SNAT       udp  --  127.0.0.11           anywhere             udp spt:51249 to:169.254.1.1:53

After Daemon restart additional rules appear, which are the default when not using sysbox.

iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER_OUTPUT  all  --  anywhere             169.254.1.1
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER_OUTPUT  all  --  anywhere             127.0.0.11
DOCKER_OUTPUT  all  --  anywhere             169.254.1.1
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER_POSTROUTING  all  --  anywhere             127.0.0.11
DOCKER_POSTROUTING  all  --  anywhere             169.254.1.1
Chain DOCKER_OUTPUT (3 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere             127.0.0.11           tcp dpt:domain to:127.0.0.11:39639
DNAT       udp  --  anywhere             127.0.0.11           udp dpt:domain to:127.0.0.11:43355
DNAT       tcp  --  anywhere             169.254.1.1          tcp dpt:domain to:127.0.0.11:46183
DNAT       udp  --  anywhere             169.254.1.1          udp dpt:domain to:127.0.0.11:51249

Since sysbox rewrites resolv.conf to use 169.254.1.1 as dns and not 127.0.0.11 dns requests breaks.

If you need any more information I am happy to provide them.
Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.