Azure / Azure/azure-container-networking

azure-cni /v2/ images install a dynamically linked azure-vnet (needs GLIBC_2.32/2.34), breaking CNI on Ubuntu 20.04 nodes during 1.34→1.35 upgrades

Open
#4,793 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
434
Forks
276
Avg merge
5d 20h
Merged PRs (30d)
33

Description

### Summary

The `azure-cni` images published under `mcr.microsoft.com/containernetworking/v2/` ship an `azure-vnet` binary that is **dynamically linked** and requires `GLIBC_2.32` and `GLIBC_2.34`. The images under the original `mcr.microsoft.com/containernetworking/` path ship a **statically linked** binary with no libc dependency.

Because `dropgz` / `cni-installer` copies this binary **out of the container and onto the host** (`/opt/cni/bin/azure-vnet`), the container's own base image is irrelevant at runtime — the node's glibc decides whether the binary can run. On Ubuntu 20.04 (glibc 2.31) it cannot, and all pod sandbox creation on that node fails:

```
plugin type="azure-vnet" failed (add): netplugin failed:
"/opt/cni/bin/azure-vnet: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found
(required by /opt/cni/bin/azure-vnet)
/opt/cni/bin/azure-vnet: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
(required by /opt/cni/bin/azure-vnet)"
```

### Why this matters on AKS

The `azure-cns` DaemonSet image is selected by the cluster's Kubernetes version, and a DaemonSet by definition rolls out to **every** node. Observed on AKS:

| Cluster Kubernetes version | `cni-installer` init container image | linkage |
|---|---|---|
| 1.34 | `containernetworking/azure-cni:v1.7.16-0` | static |
| 1.35 | `containernetworking/v2/azure-cni:v1.8.11` | dynamic (GLIBC 2.32 / 2.34) |

During a 1.34 → 1.35 upgrade the control plane is upgraded first, so the new binary is installed on nodes that are still running the old node image. Ubuntu 20.04 is still the OS for **FIPS-enabled** node pools, so those clusters end up with un-upgraded nodes whose CNI no longer works.

Already-running pods are unaffected (the CNI only runs at sandbox creation), so nothing appears broken. The failure only surfaces when a new pod is scheduled onto a not-yet-upgraded node — which is exactly what happens during the rolling node drain. Any pod that lands there stays `Pending` indefinitely, and Kubernetes does not reschedule a pod that fails sandbox creation. If that pod is a replica of a workload with a PodDisruptionBudget, the budget can then refuse eviction of its healthy sibling, and the node pool upgrade deadlocks and retries until it is abandoned.

### Reproduction (no cluster required)

Both images can be pulled anonymously from MCR:

```
# original repo — 1 layer, FROM scratch
mcr.microsoft.com/containernetworking/azure-cni:v1.7.16-0
label image.base.ref.name = scratch
dropgz: ELF 64-bit LSB executable, statically linked
embedded azure-vnet: statically linked, no libc references

# /v2/ repo — 2 layers, Azure Linux base
mcr.microsoft.com/containernetworking/v2/azure-cni:v1.8.11
dropgz: ELF 64-bit LSB executable, dynamically linked,
interpreter /lib64/ld-linux-x86-64.so.2
embedded azure-vnet: dynamically linked, libc.so.6,
required symbols: GLIBC_2.2, 2.3, 2.4, 2.32, 2.34
```

Steps: pull the linux/amd64 manifest for each tag, extract the largest layer, take `dropgz` (`/dropgz` in the old image, `/usr/bin/dropgz` in the new one) and run `file` on it. The CNI binaries themselves are embedded in `dropgz` as plain gzip streams and can be extracted by scanning for the gzip magic bytes and decompressing; running `file` and `strings | grep GLIBC_` on the resulting ELF files shows the same difference.

Note this is not a version bump: `v1.7.17` exists in **both** repos — statically linked under `containernetworking/azure-cni:v1.7.17-0`, dynamically linked under `containernetworking/v2/azure-cni:v1.7.17`. The change came with the build pipeline, not the code.

### Likely origin

This looks like a consequence of the Microsoft Go toolchain move rather than an intentional change:

- #4494 (Go 1.24 → 1.26.4) notes that MS Go 1.26 enables `systemcrypto` by default, which requires CGO, and that `GOEXPERIMENT=ms_nocgo_opensslcrypto` became mandatory for `CGO_ENABLED=0` Linux builds.
- #4673 hit the same class of problem for `ipv6-hp-bpf`, and describes it precisely: *"Go 1.26 (Microsoft fork) with `GOEXPERIMENT=ms_nocgo_opensslcrypto` dynamically loads these libraries at runtime via `dlopen()`, even with `CGO_ENABLED=0`."* That was fixed by changing the container base image.

The base-image fix works for binaries that execute **inside** the container. It does not help binaries that `dropgz` installs onto the host, where the node's own libraries are what matter. The repo's `cni.Dockerfile` still declares `FROM scratch` and the build still sets `CGO_ENABLED=0`, so the shipped `/v2/` image does not match the invariant the build config appears to intend.

### Impact

Any AKS cluster with FIPS-enabled node pools on Ubuntu 20.04 that upgrades to Kubernetes 1.35 will have broken pod networking on every node that has not yet been replaced. This is the migration path currently recommended for the Ubuntu 20.04 retirement (Azure/AKS#4874), so it is likely to be hit more as that deadline approaches.

### Expected behaviour

Either:

1. Build the CNI binaries that are installed onto the host without a runtime dependency on host libraries (as the original images did), or
2. Do not roll a CNI image onto nodes whose OS cannot execute the binary it installs — e.g. keep the DaemonSet image aligned with the node image rather than only with the control plane version.

At minimum, a documented minimum host glibc / OS version for the `/v2/` images would let operators avoid the upgrade path that triggers this.

Contributor guide

Open the contributing guide

Research direction

Start with cni.Dockerfile and the /v2/ image build, then compare the embedded binaries produced for the original and v2 image paths. Inspect the dropgz/cni-installer handoff and the findings from #4673 to determine how host-installed binaries should be built. Done means the CNI binary runs on Ubuntu 20.04 nodes without the reported GLIBC dependency, with the image contents verified using file and strings.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, kubernetes, linux
Domain
build-system, infrastructure, networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.