Mobile Data works fine, WiFi dont cant connect. work before update from a 5 month older version.

### Describe the Bug

# Bug Report: Persistent connection instability / infinite holepunch loop between Newt and Gerbil

## Summary

Client connections to a Newt site (`homelab`) fail to stabilize. Newt continuously cycles through `Triggering on-demand hole punch` → `Removing Endpoint for peer` → `Peer ... updated successfully` roughly every 1-2 seconds, indefinitely, and never settles into a stable connected state on some networks. The client (Pangolin CLI / Android / Windows app) correspondingly shows the site status flapping between `Connected`/`Disconnected` with the endpoint's NAT-mapped port changing constantly.

This occurs **independently of client platform** (Windows, Android, Linux CLI), **independently of client version** (tested CLI 0.14.0 and 0.15.1), and **independently of the physical network** the client is on (home Wi-Fi, a relative's Wi-Fi on a different ISP), but does **not** occur on mobile data, where connections establish successfully in ~8 seconds via relay.

## Symptoms observed

1. **Newt log loop (site-side), continuous, ~every 1-2s:**
```
INFO: Triggering on-demand hole punch to 1 exit nodes
INFO: Successfully sent on-demand hole punch to 1/1 exit nodes
INFO: Removing Endpoint for peer =
INFO: Peer = updated successfully
```
This loop runs indefinitely and never stabilizes into a settled WireGuard session for the affected site when the client is on the same WAN IP as that site.

2. **Pangolin server log (main app container):** No "hole punch too old" or stale-timestamp rejections observed (clock sync between server and client was verified — `timedatectl` shows `System clock synchronized: yes` on both, correct CEST offset). Server-side `Handling relay olm message!` / `Updated peer ... on newt ...` messages *do* appear consistently, suggesting the control-plane websocket coordination itself is healthy.

3. **Client status output flaps constantly:**
```
SITE ENDPOINT STATUS LAST SEEN CONNECTION
homelab public-ip:XXXXX Disconnected Ns ago Direct
vpn public-ip:XXXXX Disconnected Ns ago Direct
fars netværk his public ip:XXXXX Disconnected Ns ago Direct
```
The port number in the endpoint changes on nearly every poll.

4. **tcpdump on the client (Wi-Fi) during a connection attempt** shows repeated outbound UDP handshake/keepalive packets to the site's public IP with exponential backoff (1s, 2s, 4s, 8s, 16s, 32s...), and **zero inbound packets** in return, over a 60+ second window.

5. **OPNsense firewall live log** shows the corresponding inbound UDP packets from the VPS's Gerbil relay IP being hit with `block / Default deny / state violation rule` on the client's own router — i.e., the return traffic from Gerbil doesn't match an existing firewall state, because the outbound NAT mapping had already changed to a different ephemeral port by the time the reply arrived.

## Root causes identified during investigation

We identified **two distinct, compounding issues**:

### 1. Outbound NAT port randomization on the client-side router (OPNsense) — confirmed and fixed

OPNsense's automatic outbound NAT rule (`Static Port: NO`) was randomizing the external source port for outbound UDP on every packet/session. Combined with Newt's on-demand hole-punch retry loop (which resends every 1-2s), this meant return traffic from Gerbil/Newt's relay endpoint frequently arrived at a port OPNsense's firewall state table no longer recognized, and was dropped as `state violation`.

**Fix applied:** Added a manual outbound NAT rule in OPNsense (`Firewall → NAT → Outbound`, Hybrid mode) for the relevant source IP with `Static Port: YES`, forcing a consistent external port. This **resolved relay-based connections** — confirmed via client status output showing `connected: true, isRelay: true` for all three sites when tested from mobile data / after the fix.

### 2. Pangolin 1.21 "Same Network Detection" offering an unreachable candidate — confirmed, unresolved

Pangolin 1.21 introduced "Same Network Detection" ([announcement](https://pangolin.net/news/1-21-release), [NAT traversal deep-dive](https://pangolin.net/news/nat-holepunching)):

> "Add same network detection for clients and sites, so connection don't relay when they are on the same network (requires updated clients and sites)"
> "Pangolin detects this during candidate discovery by probing local ip:port addresses alongside public ones. When both peers are on the same LAN, they form a direct peer-to-peer connection over the local network."

In our setup, the Wi-Fi client and the `homelab`/`vpn` sites share the **same public WAN IP** (both behind the same OPNsense router). This appears to trigger "same network" detection, and Pangolin/Newt then offers a **local candidate endpoint that is not actually reachable**:

- Before adding `network_mode: host` to the Newt container: the reported local candidate was the **Docker bridge-internal IP** (`172.16.5.2`, from the default `pandolin_default` bridge network) — completely unroutable from any real client.
- After adding `network_mode: host` to Newt (per the [Install Sites docs](https://docs.pangolin.net/manage/sites/install-site) recommendation for LAN reachability): the connection loop **persisted identically**, with no observable change to the `Triggering on-demand hole punch` / `Removing Endpoint` cycle.

Mobile data clients never trigger this "same network" path (different public IP than the sites), and correctly fall back to relay — which now works reliably after fix #1 above.

## What we tried that did **not** resolve the core loop

For completeness/troubleshooting reference, the following were tested and ruled out:

- Changing/removing static DNS configuration on the client (primary/secondary upstream DNS, including matching Pangolin CLI's separate `--dns` endpoint-resolution flag vs `--upstream-dns`)
- Downgrading the Pangolin CLI client from 0.15.1 to 0.14.0 — identical failure behavior on both
- Setting `--holepunch=false` on the Pangolin CLI client — connection still reported as `Direct` and still failed to stabilize (this itself may be worth investigating: the flag did not appear to force relay-only behavior as expected)
- Adding `network_mode: host` to the Newt container for the affected site
- Verifying NTP/clock sync between server and client (both synchronized, correct timezone offset — ruled out a suspected "hole punch too old" timestamp-skew class of bug seen in other reports, e.g. fosrl/pangolin#2400, #2318, #1526, discussion #2161)
- Confirming the VPS has a genuine static public IP (not CGNAT/double-NAT) via OPNsense WAN status vs. external IP check
- Confirming UDP reachability with `tcpdump` and Gerbil-side logs simultaneously — packets do leave the client and do arrive at Gerbil (`No proxy mapping found for :` entries correlate with client-side outbound packets), but the sustained bidirectional session for the "same network" / direct path never stabilizes
- Adding an explicit `Pass` firewall rule on OPNsense WAN for inbound traffic from the Gerbil/VPS IP (in addition to the outbound static-port fix) — relay path improved, but the recurring `Removing Endpoint` loop for the affected site persisted regardless

## Request / Questions for maintainers

1. Is there a way to **disable "Same Network Detection" per-site or per-organization**? We could not find a toggle in the admin UI for Pangolin 1.21.1.
2. Is the local candidate address supposed to be the Newt container's Docker-internal bridge IP, or the host's real LAN IP obtained via `network_mode: host`? Our testing suggests neither is actually being used successfully to establish a working local/direct path, and the endpoint continues to flap even in `network_mode: host`.
3. Could the `Triggering on-demand hole punch` → `Removing Endpoint` cycle observed in Newt logs (repeating every 1-2s indefinitely, with no eventual stabilization) be a bug in how Newt handles NAT/endpoint updates when "same network" is detected but the local candidate doesn't work? Is there a way to configure it to prefer/fall back to relay when local candidates fail repeatedly, rather than looping indefinitely?
4. Does `--holepunch=false` on the Pangolin CLI client also need a corresponding site-side setting (e.g. on Newt) to fully force relay-only behavior? It did not appear to change the client's reported `Direct` connection attempt.

## Logs available on request

We have detailed, timestamped logs from the full chain (Pangolin CLI/Android/Windows client, Newt, Gerbil, and Pangolin server) covering multiple test sessions across Wi-Fi, mobile data, and a third-party network, including simultaneous `tcpdump` captures and OPNsense firewall live-log entries, and are happy to provide further detail or run additional diagnostic commands if useful.

### Environment

## Environment

- **Pangolin server:** 1.21.1 (Docker, `docker.io/fosrl/pangolin:1.21.1`)
- **Gerbil:** 1.4.3 (Docker, `docker.io/fosrl/gerbil:1.4.3`)
- **Traefik:** v3.6, running as `network_mode: service:gerbil`
- **Newt:** `fosrl/newt:latest` (resolved to Newt 1.15.0 at time of testing), running on TrueNAS SCALE via Docker
- **Router/Firewall (client side and one Newt site):** OPNsense 26.1.1-amd64
- **Client tested:** Pangolin CLI 0.14.0 and 0.15.1 (Linux/CachyOS), Pangolin Android app, Pangolin Windows client
- **Sites configured:** 3 sites total — `homelab` (same public IP as the affected Wi-Fi client), `vpn` (same public IP as the affected Wi-Fi client), `fars netværk` (different public IP, unaffected)
- **VPS public IP:** static, confirmed via OPNsense WAN interface + external IP check (not CGNAT/double-NAT)

### To Reproduce

i don't know how to get out of it, so i expect it's altway that way or is there some one with another experience ?

### Expected Behavior

i expect the tunnel to work when i'm on the same network or on a mobile data away from home.

Start with the reported Newt log loop, same-network candidate discovery, and the Pangolin CLI behavior when `--holepunch=false`; compare sessions on Wi-Fi and mobile data using the available Newt, Gerbil, client, tcpdump, and OPNsense logs. Done would mean identifying a reproducible cause and ensuring same-network failures stabilize through a reachable direct path or a relay fallback.

Written by the indexing model from the issue text.