Newt dials an unreachable IP when the target container has several networks shared with Newt
- Dominant language
- Go
- Stars
- 908
- Forks
- 87
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 7
Description
### Describe the Bug
## Issue
When a container is attached to more than one Docker network that Newt is also attached to, Newt resolves the destination hostname via Docker DNS and dials one of the returned `A` records without checking whether it is usable.
If it picks a network with `enable_icc=false`, every request fails with `dial tcp :: i/o timeout` and the client gets a 502 after ~30s, even though a valid IP exists on another shared network.
`DOCKER_ENFORCE_NETWORK_VALIDATION=true` does not prevent this, Newt still targets the unusable IP.
On my server, I have multiple internal docker bridge networks and an internet (but no ICC) bridge netwok.
This is how I encountered the issue.
## Workaround
My workaround for this has been to use unique container aliases specific to the network I want Newt and the container to use:
```yaml
networks:
bridge-newt:
aliases: [broken-service.newt]
# ....
- pangolin.private-resources.broken-service.destination=broken-service.newt
```
## Suggestion
I'd suggest adding an explicit docker network to use, like Traefik does with its labels, something like `pangolin.private-resources..docker-network`.
### Environment
- OS Type & Version: OpenMediaVault 7.7.24-7
- Pangolin Version: 1.22.0
- Edition (Community or Enterprise): CE
- Gerbil Version: 1.5.0
- Traefik Version: 3.7
- Newt Version: 1.16.0
- Client Version: 0.16.0 (Linux CLI)
### To Reproduce
I'm assuming you have a Pangolin instance running.
Example docker compose:
```yaml
name: newt-multinet-repro
networks:
bridge-newt:
driver: bridge
bridge-noicc:
driver: bridge
internal: true
driver_opts:
com.docker.network.bridge.enable_icc: "false"
bridge-something:
driver: bridge
internal: true
services:
newt:
image: fosrl/newt:1.16.0
restart: unless-stopped
environment:
- PANGOLIN_ENDPOINT=...
- NEWT_ID=...
- NEWT_SECRET=...
- DOCKER_SOCKET=/var/run/docker.sock
- DOCKER_ENFORCE_NETWORK_VALIDATION=true
- LOG_LEVEL=DEBUG
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
bridge-newt:
bridge-noicc:
# Reachable: shares only bridge-newt with newt
ok-service:
image: traefik/whoami
networks:
bridge-newt:
bridge-something:
labels:
- pangolin.private-resources.ok-service.name=OK Service
- pangolin.private-resources.ok-service.mode=http
- pangolin.private-resources.ok-service.full-domain=ok.${INTRANET_DOMAIN}
- pangolin.private-resources.ok-service.destination=ok-service
- pangolin.private-resources.ok-service.destination-port=80
- pangolin.private-resources.ok-service.scheme=http
# Unreachable: also shares bridge-noicc with newt
broken-service:
image: traefik/whoami
networks:
bridge-newt:
bridge-noicc:
bridge-something:
labels:
- pangolin.private-resources.broken-service.name=Broken Service
- pangolin.private-resources.broken-service.mode=http
- pangolin.private-resources.broken-service.full-domain=broken.${INTRANET_DOMAIN}
- pangolin.private-resources.broken-service.destination=broken-service
- pangolin.private-resources.broken-service.destination-port=80
- pangolin.private-resources.broken-service.scheme=http
```
Reproduce with the compose above `ok-service` works, `broken-service` times out.
The two containers are identical apart from `bridge-noicc`.
Newt log:
```
INFO: HTTP handler: GET / -> http://broken-service:80
ERROR: HTTP handler: upstream error (GET / -> http://broken-service:80): dial tcp 172.22.0.11:80: i/o timeout
```
The service itself is fine, `curl`ing it from another container succeeds.
### Expected Behavior
When Newt and a container share a docker network, referencing the container via its container name or service name should reach it.
Contributor guide
Research direction
Start from the HTTP handler path shown in the logs and trace how Docker DNS results are selected when Newt and the target share multiple networks. Reproduce the behavior with the provided compose setup and verify that requests choose a usable shared-network address instead of timing out on the network with disabled ICC.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100