fosrl / fosrl/newt

Registration is never retried when the endpoint DNS lookup fails in handleConnect (site stays online, no tunnel)

Open
#442 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
908
Forks
87
Avg merge
1d 15h
Merged PRs (30d)
7

Description

### Describe the Bug

If the DNS lookup of the WireGuard endpoint fails at the moment `handleConnect` runs, newt logs `Failed to resolve endpoint`, returns, and never tries again. The websocket stays connected, so Pangolin keeps the site **online**, but there is no WireGuard device and no proxy listeners — every resource on the site returns a gateway timeout until the process is restarted. `HEALTH_FILE` is not removed, so a Docker healthcheck does not notice either.

Why nothing recovers (`newt/connect.go`, `newt/tunnel.go`, same on `main`):

- `closeWgTunnel` has already stopped the ping loop before `handleConnect` runs.
- `handleConnect` calls `util.ResolveDomain(n.wgData.Endpoint)`; on error it logs, sets `regResult = "failure"` and `return`s.
- `pingWithRetry` — whose failure branch re-sends `newt/wg/register` — is only started *after* `ResolveDomain` succeeds.

We first hit this in production after a short resolver outage: the agent stayed in this state for ~90 minutes with the site showing online the whole time. Two other agents on the same network recovered because their lookups happened to succeed.

Minimal reproduction: https://github.com/bigjonroberts/newt-resolve-failure-repro

### Environment

- OS Type & Version: Linux x86_64 (Docker 29.5, WSL2 kernel 6.18); newt in the official Alpine-based container image
- Pangolin Version: 1.22.0
- Edition (Community or Enterprise): Community
- Gerbil Version: `fosrl/gerbil:latest` as of 2026-09-04 (digest `6cde03ff…`)
- Traefik Version: 3.7.13
- Newt Version: 1.16.0 (also observed on 1.14.0 in production)
- Client Version: n/a

### To Reproduce

The linked repo is a single `docker compose` stack: Pangolin, gerbil, Traefik, a `whoami` backend, newt, and a tiny controllable DNS server that is newt's only resolver. newt reaches the control plane via `/etc/hosts` (no DNS), while the WireGuard endpoint name `gerbil.repro.test` is served only by that DNS server. With one exit node newt does not HTTP-ping it, so the only lookup of that name is the one inside `handleConnect`.

```bash
git clone https://github.com/bigjonroberts/newt-resolve-failure-repro && cd newt-resolve-failure-repro
./setup.sh # stack up, admin + API key, org/site/resource/target via the Integration API, newt up
./repro.sh wedge # the bug
./repro.sh reset && ./repro.sh control # same WireGuard fault, DNS healthy → recovers
```

`wedge` does: switch DNS to `SERVFAIL`; drop newt's outbound UDP/51820 so the in-tunnel pings fail; wait for `Connection to server lost` (newt re-registers, Pangolin answers `newt/wg/connect`); wait for `Failed to resolve endpoint`; **clear both faults**; observe for 120 s.

Observed:

```
WARN Connection to server lost after 4 failures. Continuous reconnection attempts will be made.
INFO TCP listener closed, stopping proxy handler for :46085
INFO Connecting to endpoint: gerbil.repro.test
ERROR Failed to resolve endpoint: DNS lookup failed: lookup gerbil.repro.test on 127.0.0.11:53: server misbehaving
INFO Stopping ping check
<- nothing further, 126 s after DNS and UDP were restored
GET /v1/site/1 -> {"online": true}
curl -H 'Host: whoami…' edge -> no response / 504
http://100.89.128.4:46085 from the gerbil netns -> no listener
```

`control` (identical fault, DNS left working) ends with `Started tcp proxy to 172.28.0.20:80` two seconds after the UDP block is lifted. `./repro.sh startup` (DNS failing at first registration) shows the same dead end from a cold start: `Failed to resolve endpoint`, site `online`, resource 502, no retry.

### Expected Behavior

After a failed endpoint lookup, newt should keep trying — either retry `ResolveDomain` with backoff inside `handleConnect`, or treat a failed registration like a failed ping and re-enter the `newt/ping/request` → `newt/wg/register` path. It would also help if a failed registration removed `HEALTH_FILE`, so an external healthcheck can restart the agent in the meantime.

Related: #284 / #342 (ping-recovery gating), #424 (pending registration chain), #355 (holepunch re-resolution) — none cover this branch. Happy to test a fix.

Contributor guide

Open the contributing guide

Research direction

Start in newt/connect.go and newt/tunnel.go, especially handleConnect, closeWgTunnel, and pingWithRetry. Run the linked reproduction with ./repro.sh wedge and inspect the registration path after the endpoint lookup fails. Done means newt recovers after DNS and UDP are restored instead of remaining online without a WireGuard device or proxy listeners.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.