containers / containers/netavark
per-bridge sysctl.d drop-ins leaked for internal networks, disabling forwarding on unrelated external networks
- Dominant language
- Rust
- Stars
- 794
- Forks
- 161
- Avg merge
- 8d 17h
- Merged PRs (30d)
- 3
Description
I'm creating two networks in a podman compose, one internal (`internal: true`), one with default external config. I run this in a podman machine on a MacOS host. The first time around, this would always work fine. But when I repeatedly tear down the compose and re-up it, sometimes the external network would not forward my packages externally, other times it would work just fine.
## Deterministically reproducing the issue
This was all very non-deterministic, but I figured out a way to trigger the bug deterministically.
In my podman machine running under MacOS (see "My environment" below; should work on any linux hosts with systemd-sysctl and udev as well):
```console
# podman network create --internal a
a
# BR=$(podman network inspect a --format '{{.NetworkInterface}}')
# podman run -d --name c1 --network a docker.io/library/alpine sleep 60
30900c35fb544e101a9bf1eb9d5ec911d8b3b5265eeda945339c2fc994d80793
# cat /run/sysctl.d/10-netavark-$BR.conf
# autogenerated by netavark
net/ipv4/conf/podman3/forwarding = 0
net/ipv4/conf/podman3/rp_filter = 2
# podman rm -f -t0 c1 && podman network rm a
c1
a
# ls -l /run/sysctl.d/10-netavark-$BR.conf # <-- this is left lying around from the deleted network
-rw-r--r--. 1 root root 101 Aug 9 12:41 /run/sysctl.d/10-netavark-podman3.conf
# podman network create b # reuses the bridge name
b
# podman run -d --name c2 --network b docker.io/library/alpine sleep 60
e365680526e5a84ac6aa30dd543e4f2937af5d07d8ad35d1033b9d20ddb19c5a
# cat /proc/sys/net/ipv4/conf/$BR/forwarding # <-- external network, forwarding off
0
# podman exec c2 wget -qO- -T5 https://example.com # <-- external connectivity broken
wget: download timed out
```
## What seems to be going on
1. Setting up an internal network, netavark writes `/run/sysctl.d/10-netavark-.conf`
containing `net/ipv4/conf//forwarding = 0` (and `rp_filter = 2`).
2. Tearing the network down removes the bridge but leaves the drop-in in `/run/sysctl.d/`.
3. Bridge names are reused.
4. systemd reapplies the drop-in as per some udev-rule to every newly added interface of the same name
5. netavark's own configuration of the new bridge and the udev-triggered `systemd-sysctl` run are
concurrent and race. When `systemd-sysctl` wins the race `forwarding = 0` stays set.
## Expected behaviour
An external network's bridge must have `net.ipv4.conf..forwarding = 1`, regardless of what
any previously-deleted network with the same bridge name required.
## My environment
| | |
|---|---|
| Host | macOS (Apple Silicon, arm64), `podman machine` |
| VM | Fedora CoreOS 44.20260707.3.1 |
| Kernel | 7.1.3-200.fc44.aarch64 |
| podman | 6.0.2 (`b28edb9ad70ce4317dc762ee9ce0a6d081d154e9`, linux/arm64) |
| netavark | 2.0.0 |
| systemd | 259 (259.7-1.fc44) |
I doubt this is MacOS-specific; the whole topology is internal to the Linux VM in podman machine. Any rootful podman host with `systemd-sysctl` and udev should reproduce it.
## Workaround
Manually running
```sh
sysctl -w net.ipv4.conf..forwarding=1
```
fixes the issue for the currently configured network. Not a permanent fix, tearing down and re-upping it causes the bug to trigger again (sometimes; subject to race condition).
## AI use
Full disclosure: I had Claude Opus 5 help me debug this. Without it, this bug report would have ended after the first paragraph. I did write this entire report myself and I did manually verify and run everything described here, so I'm sure all of this is legit. Thanks to Opus, I also managed to build a fix for this, which I will submit a PR for shortly, when I'm done reviewing it.
Contributor guide
Research direction
Reproduce the teardown and recreation sequence, then inspect /run/sysctl.d/10-netavark-.conf alongside the bridge's forwarding value and the systemd-sysctl/udev behavior described in the report. Done means a recreated external bridge retains forwarding enabled despite a deleted internal network with the same bridge name, including across repeated runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100