[Feature] Request to expose containerd ImagePullProgressTimeout as a Configurable Node-Level Parameter in AKS
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
## Is your feature request related to a problem? Please describe.
Containerd's CRI implementation includes an `ImagePullProgressTimeout` parameter (default: **5 minutes**) that monitors byte-level progress during image pulls. If no new bytes are received within this window, containerd cancels the pull and allows kubelet to retry. This parameter **already exists in open-source containerd** ([source](https://github.com/containerd/containerd/blob/main/internal/cri/config/config.go)) and can be configured in standalone deployments.
However, **AKS does not expose this parameter** through any node-level configuration mechanism today.
When transient network disruptions occur between AKS nodes and the container registry (e.g., MCR), image pulls stall with 0 bytes transferred. Containerd must wait for the full 5-minute `ImagePullProgressTimeout` to expire before canceling the operation. In real customer scenarios, the network typically recovers within seconds to minutes, but pods remain stuck waiting for the hardcoded 5-minute timeout. Customers who require faster recovery from transient failures have no way to reduce this value on AKS.
**Real-world example:** In a customer case, AKS nodes experienced transient network disruptions to MCR (confirmed via Flow Logs and containerd logs). Image pulls stalled with 0 bytes transferred. After containerd's internal retry (triggered by TCP timeout at ~2 minutes), the pull succeeded in < 2 seconds — confirming the network had already recovered much earlier. In worse conditions, the full 5-minute timeout must expire before kubelet can re-initiate the pull. The customer had no way to tune this behavior.
## Describe the solution you'd like
Expose `ImagePullProgressTimeout` as a configurable node-level parameter in AKS, similar to how kubelet and OS parameters are already configurable today.
**Example — via containerd config file:**
```bash
az aks nodepool add \
--resource-group myRG \
--cluster-name myCluster \
--name myPool \
--containerd-config-file ./containerd-config.json
```
```json
{
"ImagePullProgressTimeout": "30s"
}
```
**Example — via Node Configuration Profile:**
```json
{
"properties": {
"agentPoolProfiles": [
{
"name": "nodepool1",
"containerdConfig": {
"imagePullProgressTimeout": "30s"
}
}
]
}
}
```
## Describe alternatives you've considered
- **Upstream containerd `ResponseHeaderTimeout`**: We have submitted a feature request to the containerd community ([containerd#13006](https://github.com/containerd/containerd/issues/13006)) to add `ResponseHeaderTimeout` to the CRI HTTP transport. This addresses the inner per-request timeout (reducing individual HTTP stall from ~2–3 minutes to ~30 seconds). However, this is an upstream change with its own timeline, and it addresses a different layer — the two improvements are **complementary**, not substitutes.
- **Modifying containerd config directly on nodes**: Not feasible in AKS managed environments, as node-level containerd configuration is managed by the platform and overwritten during upgrades.
- **Increasing kubelet image pull retry frequency**: Does not help because kubelet is blocked waiting for containerd's pull operation to complete or timeout — kubelet cannot interrupt containerd mid-pull.
## Additional context
- The parameter already exists in open-source containerd and requires no upstream changes — only AKS-level plumbing to expose it.
- Default behavior remains unchanged (5 minutes), so there is **zero breaking change risk**.
- **Suggested labels:** `feature-request`, `node-configuration`
**References:**
- [containerd CRI config — `ImagePullProgressTimeout`](https://github.com/containerd/containerd/blob/main/internal/cri/config/config.go)
- [containerd#13006 — Add ResponseHeaderTimeout](https://github.com/containerd/containerd/issues/13006)
- [AKS Custom Node Configuration](https://learn.microsoft.com/en-us/azure/aks/custom-node-configuration)
- [containerd Image Pull Progress Monitoring](https://github.com/containerd/containerd/blob/main/internal/cri/server/image_pull.go)
Contributor guide
Research direction
Read containerd's internal/cri/config/config.go and internal/cri/server/image_pull.go to understand ImagePullProgressTimeout, then review the AKS Custom Node Configuration documentation and the existing node-configuration entry points. Done means AKS accepts the parameter at node level, preserves the five-minute default, and allows the documented configuration examples to control image-pull timeout behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100