IpVlan support with sandboxed networking

### Description

Hello team 👋

## Problem

We've recently started to play around with IPVlan NICs on Kubernetes. Our use-case is to bring private network connectivity to running containers. It works pretty well with various container runtimes (runc, Kata containers, etc...) but we've been struggling to make it work on gVisor when using `--network=sandbox` (the default).

Overall, it looks like that IpVlan connectivity is not supported. From what we understand, it seems related to the way gVisor removes the IP on all NICs within the container namespace when using `--network=sandbox`. This is a similar issue to what's described in https://github.com/google/gvisor/issues/6549, but I think the incompatibility with IpVlan is a somewhat undocumented/unknown side effect of this behavior.

## Steps to reproduce

Here are two scripts to reproduce with gVisor + containerd (you can also find them in this [gist](https://gist.github.com/cyclimse/2bc012f61ad6b4188785b2567172d459#file-run-sh))

Scripts to reproduce (click to open)

```bash
# run.sh
#!/bin/bash

set -eu -o pipefail

function echoblue() {
echo -e "\033[1;34m$1\033[0m"
}

if [ "$#" -ne 1 ]; then
echo "Error: Invalid number of arguments."
echo "Usage: $0 "
exit 1
fi

RUNTIME="$1"
SUPPORTED_RUNTIMES=("runsc" "runc" "runsc-network-host")

if [[ ! " ${SUPPORTED_RUNTIMES[@]} " =~ " $RUNTIME " ]]; then
echo "Unsupported runtime: $RUNTIME"
echo "Supported runtimes: ${SUPPORTED_RUNTIMES[*]}"
exit 1
fi

# Create host networking setup using a veth pair and a network namespace.
# This allows the host to communicate with the ipvlan container, which is not
# possible with a dummy interface in L2 mode.
HOST_NS="host-ns"
VETH_HOST="veth-host"
VETH_NS="veth-ns-peer"
SERVER_IP="192.168.100.1"
SERVER_PORT="12345"

if ! ip netns list | grep -q $HOST_NS; then
echo "Creating host network namespace $HOST_NS..."
ip netns add $HOST_NS
fi

if ! ip link show $VETH_HOST >/dev/null 2>&1; then
echo "Creating veth pair $VETH_HOST <-> $VETH_NS..."
ip link add $VETH_HOST type veth peer name $VETH_NS
ip link set $VETH_NS netns $HOST_NS

echo "Configuring host side of veth..."
ip link set $VETH_HOST up

echo "Configuring namespace side of veth..."
ip netns exec $HOST_NS ip addr add $SERVER_IP/24 dev $VETH_NS
ip netns exec $HOST_NS ip link set $VETH_NS up
ip netns exec $HOST_NS ip link set lo up
else
echo "Veth pair $VETH_HOST already exists."
fi

# Our target will be a simple TCP server that listens on the host namespace.
# This will allow us to test connectivity from the container to the host.
ip netns exec $HOST_NS socat -v tcp-listen:$SERVER_PORT,reuseaddr,fork,bind=$SERVER_IP \
exec:"echo 'Hello from host namespace!'" &

# Create a dummy container to test networking
CONTAINER_NAME="ipvlan-test-dummy"

# Ensure the container name is unique
if nerdctl ps -a --filter "name=$CONTAINER_NAME" --format '{{.Names}}' | grep -q "$CONTAINER_NAME"; then
echo "Removing existing container $CONTAINER_NAME..."
nerdctl rm -f $CONTAINER_NAME || true
sleep 2 # Wait for the container to be removed
nerdctl rm $CONTAINER_NAME || true
fi

# Because of the way gVisor works, we need to create the IpVlan before we run the container.
# As such, we start by creating a network namespace for the container and setting it up with the IpVlan interface.
CONTAINER_NS="container-ns"
IPVLAN_NAME="ipvlan0"
IPVLAN_IP="192.168.100.2"

if ip netns list | grep -q $CONTAINER_NS; then
echo "Removing existing network namespace $CONTAINER_NS..."
ip netns del $CONTAINER_NS || true
sleep 2 # Wait for the namespace to be removed
fi
ip netns add $CONTAINER_NS

# Set up container networking using CNI bridge plugin
CNI_PATH="/opt/cni/bin"
NETNS_PATH="/run/netns/$CONTAINER_NS"
CONTAINER_ID="test-container-$$"

# Create the CNI configuration
cat >/tmp/bridge-config.json < support

CNI_VERSION="1.6.2"

# Ensure necessary packages are installed
if [ -z "$SOCAT_IS_INSTALLED" ] || [ -z "$RUNC_IS_INSTALLED" ]; then
echo "Installing required packages: socat, runc, wget, iproute2, iptables"
sudo apt-get update
sudo apt-get install -y socat runc
else
echo "Required packages are already installed."
fi

# Install gVisor
(
if [ -z "$GVISOR_IS_INSTALLED" ]; then
(
set -e
ARCH=$(uname -m)
URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}
wget ${URL}/runsc ${URL}/runsc.sha512 \
${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512
sha512sum -c runsc.sha512 \
-c containerd-shim-runsc-v1.sha512
rm -f *.sha512
chmod a+rx runsc containerd-shim-runsc-v1
sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin

cat >/usr/local/bin/runsc-network-host <

The `run.sh` script also showcases that adding back the IP on the IPVlan NIC after gVisor has created the sandbox fixes the connectivity issue. It's likely this isn't a proper fix to the issue, be, to be blunt, we don't know enough about the internals of gVisor networking to understand why IPVlan is not supported without this small hack.

### Is this feature related to a specific bug?

I'm not sure if this is a bug, the removal of the IP is intended behavior (documented in https://github.com/google/gvisor/issues/6549#issuecomment-914880555) but the lack of support for IPVlan might be an unintended side-effect.

### Do you have a specific solution in mind?

Here are some solutions we had in mind:

- Support IPVlan when using sandboxed networking.

- Skip IP removal from IPVlan NICs. Here's a patch that implements this behavior:

Patch (click to open)

Patch to apply on release https://github.com/google/gvisor/blob/release-20250616.0:

```golang
diff --git a/runsc/sandbox/network.go b/runsc/sandbox/network.go
index e8c804893..963b8b9d4 100644
--- a/runsc/sandbox/network.go
+++ b/runsc/sandbox/network.go
@@ -252,11 +252,14 @@ func createInterfacesAndRoutesFromNS(conn *urpc.Client, nsPath string, conf *con
return fmt.Errorf("getting link for interface %q: %w", iface.Name, err)
}
linkAddress := ifaceLink.Attrs().HardwareAddr
+ isIPVlan := ifaceLink.Type() == "ipvlan" // maybe just check if it has parent index == hostDev.Attrs().Index

// Collect the addresses for the interface, enable forwarding,
// and remove them from the host.
var addresses []boot.IPWithPrefix
for _, addr := range ipAddrs {
+ log.Debugf("interface %s has address %s", iface.Name, addr.String())
+
prefix, _ := addr.Mask.Size()
addresses = append(addresses, boot.IPWithPrefix{Address: addr.IP, PrefixLen: prefix})

@@ -271,6 +274,20 @@ func createInterfacesAndRoutesFromNS(conn *urpc.Client, nsPath string, conf *con
}
return fmt.Errorf("removing address %v from device %q: %w", addr, iface.Name, err)
}
+
+ if isIPVlan && addr.IP.To4() != nil {
+ log.Debugf("interface %s is an ipv4 with ipvlan: %s, adding address /32", iface.Name, addr.IP.String())
+
+ newAddr32 := fmt.Sprintf("%s/32", addr.IP.String())
+ newAddr, err := netlink.ParseAddr(newAddr32)
+ if err != nil {
+ return fmt.Errorf("cannot parse addr: %w", err)
+ }
+ err = netlink.AddrAdd(ifaceLink, newAddr)
+ if err != nil {
+ return fmt.Errorf("cannot add ipvlan addr %s: %w", newAddr32, err)
+ }
+ }
}

if conf.XDP.Mode == config.XDPModeNS {
```

- Add a new flag that, when used in combination with `--network=sandbox`, skips the removal of the IP from specific NICs defined in the specified network namespace like `--do-not-remove-ip-from="ipvlan0"`. This is a simlar proposal to https://github.com/google/gvisor/issues/6549 but scoped to specific NICs

Thanks a lot and have a lovely day!