Azure / Azure/AKS

[Feature] Request to expose containerd ImagePullProgressTimeout as a Configurable Node-Level Parameter in AKS

Open
#5,651 12 comments 1 reaction 0 assignees View on GitHub
feature-request Needs Attention :wave:
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

## Is your feature request related to a problem? Please describe.

Containerd's CRI implementation includes an `ImagePullProgressTimeout` parameter (default: **5 minutes**) that monitors byte-level progress during image pulls. If no new bytes are received within this window, containerd cancels the pull and allows kubelet to retry. This parameter **already exists in open-source containerd** ([source](https://github.com/containerd/containerd/blob/main/internal/cri/config/config.go)) and can be configured in standalone deployments.

However, **AKS does not expose this parameter** through any node-level configuration mechanism today.

When transient network disruptions occur between AKS nodes and the container registry (e.g., MCR), image pulls stall with 0 bytes transferred. Containerd must wait for the full 5-minute `ImagePullProgressTimeout` to expire before canceling the operation. In real customer scenarios, the network typically recovers within seconds to minutes, but pods remain stuck waiting for the hardcoded 5-minute timeout. Customers who require faster recovery from transient failures have no way to reduce this value on AKS.

**Real-world example:** In a customer case, AKS nodes experienced transient network disruptions to MCR (confirmed via Flow Logs and containerd logs). Image pulls stalled with 0 bytes transferred. After containerd's internal retry (triggered by TCP timeout at ~2 minutes), the pull succeeded in < 2 seconds — confirming the network had already recovered much earlier. In worse conditions, the full 5-minute timeout must expire before kubelet can re-initiate the pull. The customer had no way to tune this behavior.

## Describe the solution you'd like

Expose `ImagePullProgressTimeout` as a configurable node-level parameter in AKS, similar to how kubelet and OS parameters are already configurable today.

**Example — via containerd config file:**

```bash
az aks nodepool add \
--resource-group myRG \
--cluster-name myCluster \
--name myPool \
--containerd-config-file ./containerd-config.json
```

```json
{
"ImagePullProgressTimeout": "30s"
}
```

**Example — via Node Configuration Profile:**

```json
{
"properties": {
"agentPoolProfiles": [
{
"name": "nodepool1",
"containerdConfig": {
"imagePullProgressTimeout": "30s"
}
}
]
}
}
```

## Describe alternatives you've considered

- **Upstream containerd `ResponseHeaderTimeout`**: We have submitted a feature request to the containerd community ([containerd#13006](https://github.com/containerd/containerd/issues/13006)) to add `ResponseHeaderTimeout` to the CRI HTTP transport. This addresses the inner per-request timeout (reducing individual HTTP stall from ~2–3 minutes to ~30 seconds). However, this is an upstream change with its own timeline, and it addresses a different layer — the two improvements are **complementary**, not substitutes.
- **Modifying containerd config directly on nodes**: Not feasible in AKS managed environments, as node-level containerd configuration is managed by the platform and overwritten during upgrades.
- **Increasing kubelet image pull retry frequency**: Does not help because kubelet is blocked waiting for containerd's pull operation to complete or timeout — kubelet cannot interrupt containerd mid-pull.

## Additional context

- The parameter already exists in open-source containerd and requires no upstream changes — only AKS-level plumbing to expose it.
- Default behavior remains unchanged (5 minutes), so there is **zero breaking change risk**.
- **Suggested labels:** `feature-request`, `node-configuration`

**References:**
- [containerd CRI config — `ImagePullProgressTimeout`](https://github.com/containerd/containerd/blob/main/internal/cri/config/config.go)
- [containerd#13006 — Add ResponseHeaderTimeout](https://github.com/containerd/containerd/issues/13006)
- [AKS Custom Node Configuration](https://learn.microsoft.com/en-us/azure/aks/custom-node-configuration)
- [containerd Image Pull Progress Monitoring](https://github.com/containerd/containerd/blob/main/internal/cri/server/image_pull.go)

Contributor guide

Open the contributing guide

Research direction

Read containerd's internal/cri/config/config.go and internal/cri/server/image_pull.go to understand ImagePullProgressTimeout, then review the AKS Custom Node Configuration documentation and the existing node-configuration entry points. Done means AKS accepts the parameter at node level, preserves the five-minute default, and allows the documented configuration examples to control image-pull timeout behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.