[BUG] aks-log-collector causes very high disk read every hour
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
We observed sometimes and on some of our AKS nodes very high disk read. We first got periodically the Prometheus alert `NodeDiskIOSaturation`, after digging in we found that it is probably the `aks-log-collector`. We would like to know why it reads that heavily and how we can disable it.
On the problem node, we usually have a disk read less than 10 io/s. Since some time, we got every our (exactly at x:34) a high read of 2 k io/s which lasts around 15 minutes. After SSH-ing onto the node, here's what we found during the read storm
```sh
$ sudo iotop -o
```
| metric | value |
|---|---:|
| Total DISK READ | 207.58 M/s |
| Total DISK WRITE | 11.62 K/s |
| Current DISK READ | 207.58 M/s |
| Current DISK WRITE | 0.00 B/s |
| TID | PRIO | USER | DISK READ | DISK WRITE | COMMAND |
|---:|:---:|:---|---:|---:|---|
| 1299307 | be/4 | root | 207.58 M/s | 0.00 B/s | systemctl |
| 2313462 | be/4 | \ | 0.00 B/s | 0.00 B/s | nginx: worker process |
| 498 | be/4 | root | 0.00 B/s | 11.62 K/s | systemd-journald |
| 3422 | be/4 | root | 0.00 B/s | 0.00 B/s | containerd |
| 30712 | be/4 | root | 0.00 B/s | 0.00 B/s | containerd |
| 29803 | be/4 | root | 0.00 B/s | 0.00 B/s | containerd |
| etc. | | | | | |
```sh
$ cat /proc/1299307/cmdline | tr '\0' ' '
systemctl status --all -fr
$ ps -o ppid= -p 1299307
1299240
$ ps -fp 1299240
```
| UID | PID | PPID | C | STIME | TTY | TIME | CMD |
|---|---:|---:|---:|---:|---|---|---|
| root | 1299240 | 1 | 0 | 11:34 | ? | 00:00:00 | /bin/bash /opt/azure/containers/aks-log-collector.sh |
```sh
$ systemctl status 1299240
● aks-log-collector.service - AKS Log Collector
Loaded: loaded (/etc/systemd/system/aks-log-collector.service; static)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: activating (start) since Mon 2025-11-24 11:34:24 UTC; 8min ago
TriggeredBy: ● aks-log-collector.timer
Main PID: 1299240 (aks-log-collect)
Tasks: 3 (limit: 9504)
Memory: 252.0M (peak: 252.7M)
CPU: 47.444s
CGroup: /aks.slice/aks-log.slice/aks-log-collector.slice/aks-log-collector.service
├─1299240 /bin/bash /opt/azure/containers/aks-log-collector.sh
├─1299307 systemctl status --all -fr
└─1299308 zip -gumDZ deflate --fifo aks_logs.zip collect/systemctl-status.txt
Nov 24 11:34:24 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299299]: zip warning: Reading FIFO (Named Pipe): collect/lsipc.txt
Nov 24 11:34:24 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299299]: adding: collect/lsipc.txt (deflated 69%)
Nov 24 11:34:24 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299302]: zip warning: Reading FIFO (Named Pipe): collect/lsns.json
Nov 24 11:34:25 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299302]: adding: collect/lsns.json (deflated 93%)
Nov 24 11:34:25 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299240]: lspci not found, skipping.
Nov 24 11:34:25 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299240]: lsscsi not found, skipping.
Nov 24 11:34:25 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299240]: lsvmbus not found, skipping.
Nov 24 11:34:25 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299305]: zip warning: Reading FIFO (Named Pipe): collect/sysctl.txt
Nov 24 11:34:25 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299305]: adding: collect/sysctl.txt (deflated 89%)
Nov 24 11:34:25 aks-userz1-24582342-vmss00000H aks-log-collector.sh[1299308]: zip warning: Reading FIFO (Named Pipe): collect/systemctl-status.txt
```
```sh
$ head -20 /opt/azure/containers/aks-log-collector.sh
#! /bin/bash
#
# AKS Log Collector
#
# This script collects information and logs that are useful to AKS engineering
# for support and uploads them to the Azure host via a private API. These log
# bundles are available to engineering when customers open a support case and
# are especially useful for troubleshooting failures of networking or
# kubernetes daemons.
#
# This script runs via a systemd unit and slice that limits it to low CPU
# priority and 128MB RAM, to avoid impacting other system functions.
# ...
```
```sh
$ systemctl list-timers | grep aks-log-collector
Mon 2025-11-24 12:34:24 UTC 36min Mon 2025-11-24 11:34:24 UTC 23min ago aks-log-collector.timer aks-log-collector.service
$ systemctl cat aks-log-collector.timer
# /etc/systemd/system/aks-log-collector.timer
[Unit]
Description=AKS Log Collector Timer
[Timer]
OnActiveSec=0m
OnBootSec=5min
OnUnitActiveSec=60m
[Install]
WantedBy=timers.target
```
Summary of the read amounts: 200 MB/s, 2k io/s, 15 min. every one hour.
Now that we know the AKS log collector highly impacts my disk I/O and triggers Prometheus alerts, why does the heavy read happen and what can we do about it?
**To Reproduce**
I would like to emphasize that it may be difficult to reproduce the issue because we don't have this problem on all the nodes, and not all the time, sometimes not on any nodes at all.
**Environment (please complete the following information):**
- Kubernetes version 1.33.2
- Node image version __AzureLinux-V3gen2-202510.29.0__
- system or user node? More often on user nodes.
**Additional context**
I would be happy to provide more information if you need.
Contributor guide
Assessment
This issue has not been assessed yet.