containers / containers/netavark

per-bridge sysctl.d drop-ins leaked for internal networks, disabling forwarding on unrelated external networks

Open
#1,497 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
794
Forks
161
Avg merge
8d 17h
Merged PRs (30d)
3

Description

I'm creating two networks in a podman compose, one internal (`internal: true`), one with default external config. I run this in a podman machine on a MacOS host. The first time around, this would always work fine. But when I repeatedly tear down the compose and re-up it, sometimes the external network would not forward my packages externally, other times it would work just fine.

## Deterministically reproducing the issue

This was all very non-deterministic, but I figured out a way to trigger the bug deterministically.

In my podman machine running under MacOS (see "My environment" below; should work on any linux hosts with systemd-sysctl and udev as well):
```console
# podman network create --internal a
a
# BR=$(podman network inspect a --format '{{.NetworkInterface}}')
# podman run -d --name c1 --network a docker.io/library/alpine sleep 60
30900c35fb544e101a9bf1eb9d5ec911d8b3b5265eeda945339c2fc994d80793

# cat /run/sysctl.d/10-netavark-$BR.conf
# autogenerated by netavark
net/ipv4/conf/podman3/forwarding = 0
net/ipv4/conf/podman3/rp_filter = 2

# podman rm -f -t0 c1 && podman network rm a
c1
a

# ls -l /run/sysctl.d/10-netavark-$BR.conf # <-- this is left lying around from the deleted network
-rw-r--r--. 1 root root 101 Aug 9 12:41 /run/sysctl.d/10-netavark-podman3.conf

# podman network create b # reuses the bridge name
b
# podman run -d --name c2 --network b docker.io/library/alpine sleep 60
e365680526e5a84ac6aa30dd543e4f2937af5d07d8ad35d1033b9d20ddb19c5a

# cat /proc/sys/net/ipv4/conf/$BR/forwarding # <-- external network, forwarding off
0
# podman exec c2 wget -qO- -T5 https://example.com # <-- external connectivity broken
wget: download timed out
```

## What seems to be going on

1. Setting up an internal network, netavark writes `/run/sysctl.d/10-netavark-.conf`
containing `net/ipv4/conf//forwarding = 0` (and `rp_filter = 2`).
2. Tearing the network down removes the bridge but leaves the drop-in in `/run/sysctl.d/`.
3. Bridge names are reused.
4. systemd reapplies the drop-in as per some udev-rule to every newly added interface of the same name
5. netavark's own configuration of the new bridge and the udev-triggered `systemd-sysctl` run are
concurrent and race. When `systemd-sysctl` wins the race `forwarding = 0` stays set.

## Expected behaviour

An external network's bridge must have `net.ipv4.conf..forwarding = 1`, regardless of what
any previously-deleted network with the same bridge name required.

## My environment

| | |
|---|---|
| Host | macOS (Apple Silicon, arm64), `podman machine` |
| VM | Fedora CoreOS 44.20260707.3.1 |
| Kernel | 7.1.3-200.fc44.aarch64 |
| podman | 6.0.2 (`b28edb9ad70ce4317dc762ee9ce0a6d081d154e9`, linux/arm64) |
| netavark | 2.0.0 |
| systemd | 259 (259.7-1.fc44) |

I doubt this is MacOS-specific; the whole topology is internal to the Linux VM in podman machine. Any rootful podman host with `systemd-sysctl` and udev should reproduce it.

## Workaround

Manually running

```sh
sysctl -w net.ipv4.conf..forwarding=1
```

fixes the issue for the currently configured network. Not a permanent fix, tearing down and re-upping it causes the bug to trigger again (sometimes; subject to race condition).

## AI use

Full disclosure: I had Claude Opus 5 help me debug this. Without it, this bug report would have ended after the first paragraph. I did write this entire report myself and I did manually verify and run everything described here, so I'm sure all of this is legit. Thanks to Opus, I also managed to build a fix for this, which I will submit a PR for shortly, when I'm done reviewing it.

Contributor guide

Open the contributing guide

Research direction

Reproduce the teardown and recreation sequence, then inspect /run/sysctl.d/10-netavark-.conf alongside the bridge's forwarding value and the systemd-sysctl/udev behavior described in the report. Done means a recreated external bridge retains forwarding enabled despite a deleted internal network with the same bridge name, including across repeated runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.