moby / moby/libnetwork

Unable to use Unbound on host within containers

Open
#1,920 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

I saw a couple of semi-related issues, but none that addressed this issue specifically.

I am using AWS ECS, and have Unbound running on the host, with ACLs open to the world (0/0). The host is pointing to its eth0 address in resolv.conf so that Docker doesn't filter it out like with 127.0.0.1.

The issue is that DNS does not work in the container unless it's via TCP (which is not ideal for many reasons, a tough one for us is that we have containers with glibc and musl, which work differently with this).

from host:

[root@ip-192-168-78-194 ~]# cat /etc/resolv.conf
options timeout:2 attempts:5
; generated by /sbin/dhclient-script
search ec2.internal
nameserver 192.168.78.194

[root@ip-192-168-78-194 ~]# cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2017.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.03"
PRETTY_NAME="Amazon Linux AMI 2017.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

[root@ip-192-168-78-194 ~]# iptables-save
# Generated by iptables-save v1.4.18 on Fri Aug 25 01:39:51 2017
*nat
:PREROUTING ACCEPT [82277:5105155]
:INPUT ACCEPT [24443:1635100]
:OUTPUT ACCEPT [215818:14974498]
:POSTROUTING ACCEPT [215818:14974498]
:DOCKER - [0:0]
-A PREROUTING -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 127.0.0.1:51679
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 127.0.0.1:51679
-A OUTPUT -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Fri Aug 25 01:39:51 2017
# Generated by iptables-save v1.4.18 on Fri Aug 25 01:39:51 2017
*filter
:INPUT ACCEPT [3358315:1418582552]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [3974094:737683939]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
COMMIT
# Completed on Fri Aug 25 01:39:51 2017

[root@ip-192-168-78-194 ~]# docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   7392c3b/17.03.1-ce
 Built:        Tue May 30 17:59:44 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   7392c3b/17.03.1-ce
 Built:        Tue May 30 17:59:44 2017
 OS/Arch:      linux/amd64
 Experimental: false

from container:

 root@310cd923c359:/# dig google.com @192.168.78.194
;; reply from unexpected source: 172.17.0.1#53, expected 192.168.78.194#53
;; reply from unexpected source: 172.17.0.1#53, expected 192.168.78.194#53
;; reply from unexpected source: 172.17.0.1#53, expected 192.168.78.194#53

; <<>> DiG 9.10.3-P4-Ubuntu <<>> google.com @192.168.78.194
;; global options: +cmd
;; connection timed out; no servers could be reached

root@310cd923c359:/# dig google.com @192.168.78.194 +tcp

; <<>> DiG 9.10.3-P4-Ubuntu <<>> google.com @192.168.78.194 +tcp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27517
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		42	IN	A	172.217.5.238

;; Query time: 0 msec
;; SERVER: 192.168.78.194#53(192.168.78.194)
;; WHEN: Fri Aug 25 01:51:41 UTC 2017
;; MSG SIZE  rcvd: 55

A tcpdump on the host shows that with TCP, the container IP hits the host IP directly, and gets a response directly.
In UDP mode, the container IP hits the host IP directly, but the response comes from 172.17.0.1.

Do you have any clues on how I can narrow down and fix the issue?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied Docker container DNS queries against 192.168.78.194, comparing UDP and TCP with tcpdump and the host's iptables-save output. Trace the libnetwork networking path responsible for the UDP reply source; done means UDP DNS replies from the queried host address are accepted by the container and resolution works alongside TCP.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.