carvel-dev / carvel-dev/kwt

kwt net fails to start, dns doesnt resolve

Open
#21 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
166
Forks
13
PR merge metrics
No merged PRs in 30d

Description

```
kwt version
Client Version: 0.0.6

Succeeded
```

Running kwt never reaches the "ForwardingProxy: Ready" log line as in the README

```
sudo -E kwt net start --debug
02:37:21PM: debug: KubeSubnets: Finished fetching pods (53) and services (29) in 29.67472ms
02:37:21PM: debug: ReconnSSHClient: Trying to reconnect SSH client
02:37:21PM: info: KubeEntryPoint: Creating networking client secret 'kwt-net-ssh-key' in namespace 'default'...
02:37:21PM: info: KubeEntryPoint: Creating networking host secret 'kwt-net-host-key' in namespace 'default'...
02:37:21PM: info: KubeEntryPoint: Creating networking pod 'kwt-net' in namespace 'default'
02:37:21PM: info: KubeEntryPoint: Waiting for networking pod 'kwt-net' in namespace 'default' to start...
02:37:21PM: debug: KubePortForward: Starting port forwarding
02:37:21PM: debug: KubePortForward: out: Forwarding from 127.0.0.1:44959 -> 2048

02:37:21PM: debug: KubePortForward: err:
02:37:21PM: debug: ReconnSSHClient: Reconnected SSH client
02:37:21PM: info: dns.FailoverRecursorPool: Starting with '127.0.0.1:53'
02:37:21PM: debug: dns.DomainsMux: Updating DNS domain handlers: map[cluster.local.:kube-dns]
02:37:21PM: info: dns.DomainsMux: Registering cluster.local.->kube-dns
02:37:21PM: debug: dns.DNSOSCache: Skipping clearing of OS DNS cache
02:37:21PM: debug: dns.DomainsMux: Updating DNS domain handlers: map[cluster.local.:kube-dns]
02:37:21PM: info: TCPProxy: Started proxy on 127.0.0.1:45955
02:37:21PM: info: UDPProxy: Started proxy on 127.0.0.1:40387
02:37:21PM: info: dns.Server: Started DNS server on 127.0.0.1:37265 (TCP) and 127.0.0.1:38123 (UDP)
02:37:21PM: debug: OsCmdExecutor: Running 'iptables -w -L -t nat'
02:37:24PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:27PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:30PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:33PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:36PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:39PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:42PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:45PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:48PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:51PM: debug: dns.DomainsMux: Updating DNS domain handlers: map[cluster.local.:kube-dns]
02:37:51PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:54PM: debug: SSHClient: Sending keepalive: false [] %!s()
02:37:57PM: debug: SSHClient: Sending keepalive: false [] %!s()

```

also dns lookups fail

```
> dig whoami.demo.svc.cluster.local

; <<>> DiG 9.11.5-P4-5.1ubuntu2.1-Ubuntu <<>> whoami.demo.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 21784
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;whoami.demo.svc.cluster.local. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Mar 13 14:42:05 AEDT 2020
;; MSG SIZE rcvd: 58
```

```
> kwt net service -n demo

Services in namespace 'demo'

Name Internal DNS Cluster IP Ports
netshoot-headless netshoot-headless.demo.svc.cluster.local None 80/tcp
whoami whoami.demo.svc.cluster.local 10.103.36.93 80/tcp
whoami-external-name whoami-external-name.demo.svc.cluster.local - -

3 services

Succeeded
```

There are no logs in the kwt-net pod

```
> kubectl -n default get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kwt-net 1/1 Running 0 24m 10.244.0.13 kind-control-plane
> kubectl -n default logs kwt-net

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing `sudo -E kwt net start --debug` and compare its output with the README's expected `ForwardingProxy: Ready` line. Inspect the networking startup and DNS paths implicated by the logs, then verify `dig whoami.demo.svc.cluster.local` and `kwt net service -n demo`. Done means `kwt net start` reaches the ready state and the Kubernetes service name resolves successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
cli, devops, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.