Azure / Azure/AKS

[BUG] Disk I/O performance degredation after upgrade from AKS 1.34.2 to 1.35.5 (kernel update 5.15.0-1111-azure to 6.8.0-1059-azure)

Open
#5,849 9 comments 3 reactions 1 assignee Claimed by @djsly View on GitHub
action-required bug nodepools stale
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe the bug**

After an AKS node-image upgrade that changed the Ubuntu kernel from **5.15.0-azure** to **6.8.0-1059-azure** (AKS 1.34.2 → 1.35.5), per-IO **write latency (`w_await`) on Azure Premium SSD data disks roughly doubled**, and average queue depth (`aqu-sz`) increased ~90%, for the same workload. Write IOPS, write throughput, average write size and device utilization were all flat-to-lower across the change, and read latency was unaffected. The only change was the node image/kernel.

This is observable directly from `node_exporter` metrics in Prometheus; no specific application is required to see it.

Suggested labels: `Linux`, `node pool`, `kernel`.

**To Reproduce**

Using `node_exporter` (kube-prometheus-stack / standard AKS Prometheus setup):

1. Find the kernel-upgrade time on the cluster:
```promql
count(kube_node_info) by (kernel_version)
```
Identify a contiguous window **before** the upgrade (all nodes on `5.15.0-*-azure`) and an equal-length window **after** it (all nodes on `6.8.0-1059-azure`). Use the **same time-of-day** for both windows to keep the workload comparable (e.g. 00:00–13:00 local, day before vs. day after).

2. Evaluate each query below twice — once with the instant-query `time` set to the **end of the BEFORE window**, once at the **end of the AFTER window** — using a range matching the window length (`[13h]` in the examples). `sd[b-z]` selects attached data disks and excludes the OS disk `sda`.

```promql
# Write await (µs per IO) -- primary signal
1e6 * sum(increase(node_disk_write_time_seconds_total{device=~"sd[b-z]"}[13h]))
/ sum(increase(node_disk_writes_completed_total{device=~"sd[b-z]"}[13h]))

# Average queue depth (aqu-sz)
sum(increase(node_disk_io_time_weighted_seconds_total{device=~"sd[b-z]"}[13h])) / (13*3600)

# Read await (µs per IO)
1e6 * sum(increase(node_disk_read_time_seconds_total{device=~"sd[b-z]"}[13h]))
/ sum(increase(node_disk_reads_completed_total{device=~"sd[b-z]"}[13h]))

# Write IOPS
sum(increase(node_disk_writes_completed_total{device=~"sd[b-z]"}[13h])) / (13*3600)

# Write throughput (MB/s)
sum(increase(node_disk_written_bytes_total{device=~"sd[b-z]"}[13h])) / (13*3600) / 1e6

# Average write size (KB)
sum(increase(node_disk_written_bytes_total{device=~"sd[b-z]"}[13h]))
/ sum(increase(node_disk_writes_completed_total{device=~"sd[b-z]"}[13h])) / 1024

# Device utilization (%)
100 * avg(increase(node_disk_io_time_seconds_total{device=~"sd[b-z]"}[13h])) / (13*3600)
```

3. Compare BEFORE vs AFTER. Write await and queue depth roughly double, while IOPS, throughput, write size and utilization are unchanged or lower.

**Observed results (production cluster, matched 13 h windows day-over-day)**

| node_exporter metric | exact metric name(s) | 5.15.0-azure | 6.8.0-1059-azure | Δ |
|---|---|--:|--:|--:|
| Write await | `node_disk_write_time_seconds_total` ÷ `node_disk_writes_completed_total` | 6 538 µs | 13 625 µs | +108 % |
| Avg queue depth (aqu-sz) | `node_disk_io_time_weighted_seconds_total` | 70.0 | 131.2 | +87 % |
| Read await | `node_disk_read_time_seconds_total` ÷ `node_disk_reads_completed_total` | 1 163 µs | 1 091 µs | −6 % |
| Write IOPS | `node_disk_writes_completed_total` | 9 540 /s | 9 099 /s | −4.6 % |
| Write throughput | `node_disk_written_bytes_total` | 255.6 MB/s | 250.0 MB/s | −2.2 % |
| Avg write size | `node_disk_written_bytes_total` ÷ `node_disk_writes_completed_total` | 26.2 KB | 26.8 KB | +2.5 % |
| Device utilization | `node_disk_io_time_seconds_total` | 2.68 % | 2.55 % | −4.9 % |

**Expected behavior**

For the same workload on the same Premium SSD SKU, write await and queue depth should remain comparable across the 5.15 → 6.8 kernel upgrade, consistent with the unchanged IOPS, throughput, write size and utilization.

**Screenshots**

N/A — metric tables above.

**Environment (please complete the following information):**

- AKS Kubernetes version: **1.34.2 → 1.35.5**
- Node OS / image: Ubuntu (AKSUbuntu)
- Kernel: **5.15.0-1111-azure / 5.15.0-1114-azure → 6.8.0-1059-azure**
- Node pool: Linux, `Standard_` (please fill in)
- Data disk: **Azure Premium SSD**, attached as SCSI (`/dev/sd*`)
- Region / zones: ``, zonal node pools

**Additional context**

The regression is isolated to write-side latency/queueing; reads are unaffected, and it is not explained by load (IOPS, throughput, write sizel or lower in the AFTER window).

We saw similar degredation when we switched from Ubuntu (Kernel 5.x) to Azure Linux (Kernel 6.x) a while ago. Therfore we rolled back to Ubuntu node image but now the Kernel update arrived there too. See: https://github.com/Azure/AKS/issues/5231

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.