canonical / canonical/cloudflared-operators
charmed-cloudflared: tunnel never starts on Ubuntu Core — /etc layout masks resolv.conf, DNS falls back to localhost
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 2
Description
## Summary
On Ubuntu Core the snap cannot resolve DNS, so `cloudflared` exits immediately and systemd restart-loops the service until it gives up. Host DNS is healthy throughout — only the snap is affected.
## Environment
- `charmed-cloudflared` 2026.5.0 (rev 77), `latest/stable`, arm64
- Ubuntu Core 26 (base `core26`), snapd 2.76, Raspberry Pi 5
- Host resolution works: `resolvectl query snapcraft.io` succeeds; systemd-resolved stub on `127.0.0.53`
## Symptom
```
ERR edge discovery: error looking up Cloudflare edge IPs: the DNS query failed
error="lookup _v2-origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:45135->[::1]:53: read: connection refused"
ERR Initiating shutdown error="Could not lookup srv records on _v2-origintunneld._tcp.argotunnel.com: ..."
subprocess.CalledProcessError: Command '['/snap/charmed-cloudflared/77/usr/bin/cloudflared', 'tunnel', '--no-autoupdate', '--metrics', '127.0.0.1:39000', '--protocol', 'http2', 'run']' returned non-zero exit status 1.
systemd[1]: snap.charmed-cloudflared.cloudflared.service: Start request repeated too quickly.
```
## Cause
`charmed-cloudflared-snap/snap/snapcraft.yaml` declares (still present at HEAD):
```yaml
layout:
/etc:
bind: $SNAP_DATA/etc
```
This replaces the whole of `/etc` inside the snap's mount namespace with `$SNAP_DATA/etc`, which is empty on a fresh install. The base's `/etc/resolv.conf` (a symlink to `../run/systemd/resolve/stub-resolv.conf`) is therefore masked:
```
$ sudo snap run --shell charmed-cloudflared.cloudflared -c 'cat /etc/resolv.conf'
cat: /etc/resolv.conf: No such file or directory
$ sudo snap run --shell charmed-cloudflared.cloudflared -c 'stat -c %n /run/systemd/resolve/stub-resolv.conf'
/run/systemd/resolve/stub-resolv.conf
```
The symlink target is reachable inside the namespace; only `/etc/resolv.conf` itself is missing.
`cloudflared` is a Go binary, and with no `/etc/resolv.conf` Go's resolver falls back to `localhost:53`. On Ubuntu Core nothing listens there — systemd-resolved binds `127.0.0.53` and `127.0.0.54` — so every lookup fails with `connection refused`, including the SRV lookup for edge discovery.
Nothing in the snap creates the file: the only hook is `configure`, and the sole reference to `resolv` anywhere in the snap is inside the `cloudflared` binary itself.
```
$ ls -A /var/snap/charmed-cloudflared/current/etc/ # empty on a fresh install
$ ls /snap/charmed-cloudflared/current/meta/hooks/
configure
```
For comparison, a snap on a `core22` base without an `/etc` layout (e.g. `node-red`) sees the symlink normally and resolves fine on the same host, which is what narrowed this down to the layout rather than to confinement or to Ubuntu Core itself.
## Workaround
Create the file the layout leaves missing:
```
printf 'nameserver 127.0.0.53\nnameserver 1.1.1.1\nnameserver 1.0.0.1\n' \
| sudo tee /var/snap/charmed-cloudflared/current/etc/resolv.conf
sudo snap restart charmed-cloudflared
```
The tunnel then registers all four connections normally and survives reboots.
## Suggested fix
Any of:
1. Seed a default `resolv.conf` into `$SNAP_DATA/etc` from an install hook — e.g. a symlink to `/run/systemd/resolve/stub-resolv.conf`, or a small file with the stub plus a public fallback.
2. Narrow the layout to the specific paths the snap needs under `/etc` rather than binding all of `/etc`.
3. If the current behaviour is intended, document that `$SNAP_DATA/etc/resolv.conf` must be provided by the operator.
Happy to test a fix on Ubuntu Core.
## Still current
Checked against `main`: `charmed-cloudflared-snap/snap/snapcraft.yaml` still carries the `/etc` layout, and `charmed-cloudflared-snap/snap/hooks/configure` only normalises snap options and restarts the service — nothing creates `$SNAP_DATA/etc/resolv.conf`.
Contributor guide
Research direction
Start with charmed-cloudflared-snap/snap/snapcraft.yaml and snap/hooks/configure, then reproduce the missing /etc/resolv.conf with the snap shell command on Ubuntu Core. Done means a fresh installation provides usable DNS inside the snap and the cloudflared service starts without restart-looping; verify that the tunnel registers and survives a restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, ubuntu
- Domain
- devops, networking, operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100