[Feature] Make Multi-Gen LRU (MGLRU) configurable
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
Yes. Our workload is suffering significant performance degradation on AKS node pools with Multi-Gen LRU (MGLRU) enabled, and there is currently no supported way to turn it off.
MGLRU (`/sys/kernel/mm/lru_gen/enabled`) is enabled by default in the kernels shipped with recent AKS node images (Ubuntu 22.04/24.04, Azure Linux). Our application can't yet leverage MGLRU's reclaim model and regresses badly under it — compared to the classic active/inactive LRU we see materially worse memory-reclaim behavior (higher reclaim CPU, latency spikes, and changed OOM timing) under the same load. We need to disable MGLRU on the affected node pools to restore acceptable performance, but every supported avenue is a dead end:
- MGLRU is controlled through a **sysfs** knob (`/sys/kernel/mm/lru_gen/enabled`) or the `lru_gen.enabled=0` kernel boot parameter — **not a sysctl**.
- The `customLinuxOSConfig` / `--linux-os-config` API only accepts a fixed set of options: `sysctls` (a hard-coded `SysctlConfig` allow-list), `transparentHugePageEnabled`, `transparentHugePageDefrag`, `swapFileSizeMB`, and `ulimitConfig`. There is no `sysfs` field and no MGLRU knob (confirmed in AgentBaker `pkg/agent/datamodel` `CustomLinuxOSConfig`/`SysctlConfig`).
- AKS doesn't expose GRUB / kernel boot-parameter customization on managed node images, so `lru_gen.enabled=0` isn't reachable either.
The result: to mitigate a real, ongoing performance problem, we're forced into an out-of-band privileged workaround (see alternatives) that we have to build, secure, and maintain ourselves — with no lifecycle guarantees from AKS.
*Label suggestion: `feature-request`, `azure/node` (node pool / OS configuration area).*
**Describe the solution you'd like**
A supported, per-node-pool toggle to disable (or enable) MGLRU, applied and re-applied by AKS during provisioning, node-image upgrades, and scale-out — the same lifecycle guarantees as existing custom node config.
Preferred: extend `customLinuxOSConfig` with an explicit, typed field rather than a generic passthrough, e.g.:
```json
{
"customLinuxOSConfig": {
"multiGenLRUEnabled": false
}
}
```
Surfaced consistently across the stack:
- `az aks nodepool add/update --linux-os-config` (and cluster create)
- ARM/Bicep `LinuxOSConfig`
- Terraform `azurerm_kubernetes_cluster(_node_pool)` `linux_os_config`
AKS would translate this to writing `/sys/kernel/mm/lru_gen/enabled` (and/or the boot parameter) on the nodes, persisted across reboots and image upgrades.
**Describe alternatives you've considered**
- **Privileged DaemonSet** that runs on every node and writes `0` to `/sys/kernel/mm/lru_gen/enabled` via `nsenter` into the host namespace. This is the approach Azure's own docs point to ("use a daemon set to customize your needed configurations without losing AKS support"), and it works and re-applies on new nodes. Downsides: it's a privileged pod we own and secure, there's a race window between node ready and the setting being applied, it clutters `kube-system`, and it's not declarative infra. It's a workaround, not a feature.
- **Generic sysfs passthrough** in `customLinuxOSConfig` (e.g. a `sysfs: [{name, value}]` list). More flexible, but broad and risky — arbitrary sysfs writes are a big support/stability surface. A single typed `multiGenLRUEnabled` flag is safer and easier to validate/support.
- **Kernel boot parameter `lru_gen.enabled=0`** — not possible on managed AKS node images (no GRUB customization exposed).
- **Custom node image** — heavy, defeats the point of managed node images and AKS-managed upgrades.
**Additional context**
- AWS EKS option https://docs.aws.amazon.com/linux/al2023/ug/kernel-mglru-al2023.html
- Kernel docs: https://docs.kernel.org/admin-guide/mm/multigen_lru.html (`/sys/kernel/mm/lru_gen/enabled` is a bitmask; writing `0` disables all MGLRU features).
- AKS custom node configuration (shows the supported subset; no MGLRU/sysfs): https://learn.microsoft.com/azure/aks/custom-node-configuration
- Related precedent: `transparentHugePageEnabled`/`transparentHugePageDefrag` are already exposed as typed OS-level toggles in `customLinuxOSConfig`, so MGLRU would fit the same pattern.
- A read-back / status of the effective value on nodes (e.g. surfaced in node labels or `az aks nodepool show`) would help validate rollout.
Contributor guide
Assessment
This issue has not been assessed yet.