Azure / Azure/AKS

[BUG] Azure Linux 3 Node Image containerd config uses deprecated v1 API path causing ACR authentication failures

Open
#5,909 3 comments 0 reactions 1 assignee Claimed by @djsly View on GitHub
action-required azurelinux bug nodepools
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe the bug**

Azure Linux 3 node pools (Node Image `202607.20.0`) fail to pull private images from Azure Container Registry (ACR) with `401 Unauthorized` errors.

**Root cause**: `/etc/containerd/config.toml` on Azure Linux 3 nodes uses the **deprecated containerd v1 API path** (`plugins.cri.v1.images`) instead of the correct containerd v2 path (`plugins.grpc.v1.cri`). This causes containerd v2.2.4 to ignore the registry configuration, preventing kubelet credential provider from being invoked for ACR authentication.

**To Reproduce**

Steps to reproduce the behavior:

1. Create an Azure Linux 3 node pool:
```bash
az aks nodepool add \
--resource-group \
--cluster-name \
--name testpool \
--node-count 1 \
--node-vm-size Standard_D4s_v3 \
--os-sku AzureLinux \
--kubernetes-version 1.33.0
```

2. Deploy a pod with a private ACR image:
```bash
kubectl run test-acr --image=.azurecr.io/your-private-image:latest \
--overrides='{"spec":{"nodeSelector":{"agentpool":"testpool"}}}'
```

3. Check pod status and events:
```bash
kubectl get pod test-acr
kubectl describe pod test-acr | grep -A5 "Events:"
```

4. Observe error: `Failed to pull image: rpc error: code = Unknown desc = failed to pull and unpack image: failed to resolve reference: pulling from host .azurecr.io failed with status code [manifests latest]: 401 Unauthorized`

**Expected behavior**

Pod should pull private ACR images successfully. The kubelet credential provider (`acr-credential-provider`) should automatically handle ACR authentication without requiring `imagePullSecrets`.

This works correctly on:
- Ubuntu node pools (same AKS version)
- Older Azure Linux node images (e.g., 202607.02.0)

**Screenshots**

N/A - This is a configuration issue visible in logs and config files.

**Environment (please complete the following information):**

- **CLI Version**: 2.68.0
- **Kubernetes version**: 1.33.0, 1.33.1 (affects both)
- **CLI Extension version**: aks-preview 10.0.0b8
- **Node OS**: Azure Linux 3.0 (CBL-Mariner)
- **Node Image Version**: 202607.20.0
- **containerd Version**: 2.2.4
- **Affected SKUs**: All VM sizes with Azure Linux 3 (tested on Standard_NC48ads_A100_v4, Standard_D4s_v3)

**Additional context**

### Root Cause: Deprecated containerd v1 API Path

The node image uses an **incorrect containerd configuration path**. On affected Azure Linux 3 nodes (`/etc/containerd/config.toml`):

**Current (broken)**:
```toml
version = 2
[plugins."io.containerd.cri.v1.images"] ← v1 API (deprecated)
[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/containerd/certs.d"
```

**Expected (working on older images)**:
```toml
version = 2
[plugins."io.containerd.grpc.v1.cri"] ← v2 API (correct)
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
```

containerd v2.x **cannot read** the deprecated `plugins.cri.v1.*` path, causing it to ignore registry configuration and skip the kubelet credential provider.

### Cross-Environment Testing

Tested on two clusters to isolate the issue:

| Environment | Node Image | containerd | Config Path | ACR Pull |
|-------------|------------|------------|-------------|----------|
| Cluster A | 202607.02.0 | 2.2.4 | `plugins.grpc.v1.cri` ✅ | ✅ Success |
| Cluster B | 202607.20.0 | 2.2.4 | `plugins.cri.v1.images` ❌ | ❌ 401 Error |

Both clusters have identical:
- containerd version (2.2.4)
- ACR credential provider binary (`/opt/azure/containers/bin/acr-credential-provider`)
- kubelet credential config (`/var/lib/kubelet/credential-provider-config.yaml`)

**Only difference**: containerd config API path

### Verification on Affected Node

```bash
# Check node image version
cat /etc/os-release
# NAME="Microsoft Azure Linux"
# VERSION="3.0.20240807"

# Check containerd config
cat /etc/containerd/config.toml | grep -A3 "plugins.*images"
# [plugins."io.containerd.cri.v1.images"] ← Wrong: v1 API

# Verify containerd version
containerd --version
# containerd github.com/containerd/containerd/v2 v2.2.4
```

### Workarounds

**Temporary fix** (lost on node reboot):
```bash
# On affected node (via kubectl debug)
cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
sed -i 's/io\.containerd\.cri\.v1\.images/io.containerd.grpc.v1.cri/g' /etc/containerd/config.toml
systemctl restart containerd
```

**Production workaround** (use `imagePullSecrets`):
```bash
kubectl create secret docker-registry acr-secret \
--docker-server=.azurecr.io \
--docker-username= \
--docker-password=
```

### Requested Fix

Update the containerd config template in Azure Linux 3 node image build pipeline to use containerd v2-compatible API path:

```diff
-[plugins."io.containerd.cri.v1.images"]
+[plugins."io.containerd.grpc.v1.cri"]
```

This aligns with:
- [containerd v2 migration guide](https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md)
- Existing working configuration in Node Image 202607.02.0
- Kubernetes best practices for containerd v2

### Impact

- **Affected**: All Azure Linux 3 node pools (Node Image 202607.20.0+)
- **Scope**: All private ACR image pulls fail without `imagePullSecrets`
- **Urgency**: Medium (workaround available, but breaks default ACR integration)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.