[BUG] Managed nodes have enormous 64MiB readahead configured for NVMe disks, which causes severe performance issues for random read heavy workloads
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
AKS managed nodes of v6 and v7 VM types with NVMe disks (PVCs and/or OS disk) have readahead (RA) (https://www.man7.org/linux/man-pages/man2/readahead.2.html) value set to `65536KiB` (can be optimal only for highly sequential read workloads - not a standard workload; Ubuntu and other distos default to `128KiB` or `4096KiB`).
This high value (50x greater than Ubuntu's defaults) can cause severe performance issues for workloads, which rely heavily on random reads such as databases, e.g. elasticsearch (https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/search-speed#_avoid_page_cache_thrashing_by_using_modest_readahead_values_on_linux).
The `65536` KiB value was introduced recently, but we cannot point it to exact date. Our workloads noticed the change on `14.04.2026` after automatic AKS node pool upgrade - this change increased the disk read throughput from a few `MiB/s` to hundreds, up to the VM limits of `460MiB` (we brought read throughput back to normal levels by decreasing RA on PVCs to 128KiB by using custom hacky daemonset and udev rules).
**To Reproduce**
1. Create AKS node pool of v7 VM (e.g. Standard_D8as_v7) with managed OS disk (NVMe).
3. (Optional, since it also applies to OS disks) Create test pod with attached PVC of `managed-csi-premium` storage class (also tested with standard SSDs, but I have to pick one in the example) and run it on newely created node pool.
4. Get shell on node, where pod is running `kubectl debug node/aks-nodename-xx-vmssx -it --image=debian:stable-slim --profile=sysadmin`
5. Check readahead values (RA column) on node `lsblk -o NAME,RA,SIZE`. For us it is:
```
lsblk -o NAME,RA,SIZE
NAME RA SIZE
sr0 128 638K
nvme0n1 65536 64G
|-nvme0n1p1 65536 63G
|-nvme0n1p14 65536 4M
|-nvme0n1p15 65536 106M
`-nvme0n1p16 65536 913M
nvme0n2 65536 8G
```
lsblk reports 65536 of RA for NVMe, both - OS and PVC disks.
**Expected behavior**
We expect RA value to be set to lower, more versatile value as what Linux distributions set (`128KiB`or even `4096`), since the current value (65536) is optimal only for small subset of workload types, whereas horribly suboptimal for others.
**Environment:**
- Kubernetes version 1.35.1
- Node pool of `Standard_D8as_v7`
**Additional context**
I believe, that the AKS nodes are configured (in most places) for general use, not for narrow use-case plane. Only very specific workloads should require tuning (especially kernel level). Besides that, the RA cannot be tuned using standard AKS or k8s configs, since azuredisk CSI (for PVCs) currently supports such feature only in alpha and OS disk cannot be tuned at all (besides hacky daemonset and udev rules approaches we used).
Contributor guide
Assessment
This issue has not been assessed yet.