Unable to use Unbound on host within containers
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 875
- PR merge metrics
- No merged PRs in 30d
Description
I saw a couple of semi-related issues, but none that addressed this issue specifically.
I am using AWS ECS, and have Unbound running on the host, with ACLs open to the world (0/0). The host is pointing to its eth0 address in resolv.conf so that Docker doesn't filter it out like with 127.0.0.1.
The issue is that DNS does not work in the container unless it's via TCP (which is not ideal for many reasons, a tough one for us is that we have containers with glibc and musl, which work differently with this).
from host:
[root@ip-192-168-78-194 ~]# cat /etc/resolv.conf
options timeout:2 attempts:5
; generated by /sbin/dhclient-script
search ec2.internal
nameserver 192.168.78.194
[root@ip-192-168-78-194 ~]# cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2017.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.03"
PRETTY_NAME="Amazon Linux AMI 2017.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
[root@ip-192-168-78-194 ~]# iptables-save
# Generated by iptables-save v1.4.18 on Fri Aug 25 01:39:51 2017
*nat
:PREROUTING ACCEPT [82277:5105155]
:INPUT ACCEPT [24443:1635100]
:OUTPUT ACCEPT [215818:14974498]
:POSTROUTING ACCEPT [215818:14974498]
:DOCKER - [0:0]
-A PREROUTING -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 127.0.0.1:51679
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 127.0.0.1:51679
-A OUTPUT -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -d 169.254.170.2/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Fri Aug 25 01:39:51 2017
# Generated by iptables-save v1.4.18 on Fri Aug 25 01:39:51 2017
*filter
:INPUT ACCEPT [3358315:1418582552]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [3974094:737683939]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
COMMIT
# Completed on Fri Aug 25 01:39:51 2017
[root@ip-192-168-78-194 ~]# docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: 7392c3b/17.03.1-ce
Built: Tue May 30 17:59:44 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: 7392c3b/17.03.1-ce
Built: Tue May 30 17:59:44 2017
OS/Arch: linux/amd64
Experimental: false
from container:
root@310cd923c359:/# dig google.com @192.168.78.194
;; reply from unexpected source: 172.17.0.1#53, expected 192.168.78.194#53
;; reply from unexpected source: 172.17.0.1#53, expected 192.168.78.194#53
;; reply from unexpected source: 172.17.0.1#53, expected 192.168.78.194#53
; <<>> DiG 9.10.3-P4-Ubuntu <<>> google.com @192.168.78.194
;; global options: +cmd
;; connection timed out; no servers could be reached
root@310cd923c359:/# dig google.com @192.168.78.194 +tcp
; <<>> DiG 9.10.3-P4-Ubuntu <<>> google.com @192.168.78.194 +tcp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27517
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 42 IN A 172.217.5.238
;; Query time: 0 msec
;; SERVER: 192.168.78.194#53(192.168.78.194)
;; WHEN: Fri Aug 25 01:51:41 UTC 2017
;; MSG SIZE rcvd: 55
A tcpdump on the host shows that with TCP, the container IP hits the host IP directly, and gets a response directly.
In UDP mode, the container IP hits the host IP directly, but the response comes from 172.17.0.1.
Do you have any clues on how I can narrow down and fix the issue?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied Docker container DNS queries against 192.168.78.194, comparing UDP and TCP with tcpdump and the host's iptables-save output. Trace the libnetwork networking path responsible for the UDP reply source; done means UDP DNS replies from the queried host address are accepted by the container and resolution works alongside TCP.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100