Azure / Azure/local-csi-driver
fsck OOM-crashloop on csi-local-node after AKS node upgrade (10 TiB ext4, 600 Mi limit)
- Dominant language
- Go
- Stars
- 33
- Forks
- 16
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 28
Description
## What happened
After an AKS node-image upgrade, the `csi-local-node` DaemonSet's `driver` container is OOM-killed in a tight crashloop while staging a previously formatted local volume. The pod consuming the PVC stays in `ContainerCreating` indefinitely with `MountVolume.MountDevice failed ... context deadline exceeded`.
The volume is ~10 TiB of ext4 holding many small files (high inode count). Each pod restart re-runs `fsck` from the top, hits the 600 Mi limit again, and gets killed. We could not break the loop without intervention.
**Mitigation we used:** delete and recreate the PVC (data was cache and could be rebuilt). We don't believe this should be the only option.
## Why it happens
`internal/csi/mounter/mounter.go` embeds `k8s.io/mount-utils.SafeFormatAndMount`, whose `formatAndMountSensitive` flow calls `checkAndRepairFilesystem` -> `fsck -a` synchronously before mount on any already-formatted rw volume ([upstream code](https://github.com/kubernetes/mount-utils/blob/master/mount_linux.go#L530)). For ext4, `fsck -a` dispatches to `e2fsck -p` (preen mode). On a clean unmount this is effectively a no-op, but after an ungraceful detach during the node upgrade the filesystem state was dirty and `e2fsck` entered repair mode.
Peak `e2fsck` memory scales with the working set of inode bitmaps, the `icount` structures (pass 1), and `dx_dir` / orphan-list state (pass 2). For a 10 TiB ext4 with the default `bytes-per-inode = 16384` (~640M inodes, [`mke2fs.conf` defaults](https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/plain/misc/mke2fs.conf.in)) holding millions of small files, repair-mode RSS routinely exceeds the 600 Mi container limit set in `charts/latest/values.yaml`:
```yaml
driver:
limits:
memory: 600Mi
requests:
cpu: 10m
memory: 60Mi
```
The fact that e2fsprogs ships an e2fsck.conf(5) scratch_files option specifically to spill these structures to disk on memory-constrained systems is itself evidence that this working set can grow well past sub-gigabyte limits on large filesystems. Azure/AKS#4682 has an independent report of the same OOM on a 12,500 GiB azuredisk volume against the same 600 Mi node-DaemonSet limit.
Two compounding factors turn the OOM into an unbreakable crashloop:
1. No timeout on fsck. SafeFormatAndMount.checkAndRepairFilesystem calls mounter.Exec.Command("fsck", "-a", source).CombinedOutput() with no context.Context, no timer, and no semaphore - unlike the format path right next to it which does honour mounter.formatTimeout. Combined with the OOM kill, every restart re-enters fsck from scratch and the loop never terminates.
2. No skip-fsck escape hatch at the mount-utils layer or as a CSI driver flag / StorageClass option. Once you're in this state, the only way out is recreating the volume or running e2fsck out-of-band on the host.
Prior art
This pattern has been reported against multiple CSI drivers and is not yet fixed upstream:
- Azure/AKS#4682 - azuredisk-csi-node OOM-killed during fsck (12,500 GiB volume, 600 Mi limit; closed-as-stale in June 2025 without a code fix).
- Azure/AKS#4421 - earlier related thread on memory pressure in the same node DaemonSet.
Repro
1. Provision a local.csi.acstor.io PVC, ~10 TiB, ext4.
2. Fill it with many small files (push inode count into the hundreds of millions).
3. Trigger an ungraceful node detach (AKS node-image upgrade, or reboot -f on the node).
4. After the node comes back, observe the csi-local-node driver container OOMKilled with the last log line being Checking for issues with fsck on disk: /dev/....
5. Pod consuming the PVC stays in ContainerCreating; kubectl get events shows MountVolume.MountDevice failed ... DeadlineExceeded.
Expected behavior
Any one of:
- The node DaemonSet has enough memory headroom to complete e2fsck -p on realistic large volumes; or
- There is a documented, supported way to skip or bound the in-line fsck (per-StorageClass parameter, per-PV annotation, or driver flag); or
- The OOM does not produce an unbreakable crashloop - e.g. fsck timeout + the driver fails the NodeStageVolume RPC cleanly with a clear error message so the operator can intervene.
Environment
- Kubernetes version: 1.35.0
- local-csi-driver version: v0.2.10 (we have not retested on v0.2.14 though nothing has changed there)
- Volume size / fs type / approx file count: ~10 TiB / ext4 / millions of small files
Contributor guide
Research direction
Start with internal/csi/mounter/mounter.go and the SafeFormatAndMount checkAndRepairFilesystem flow described in the report; compare it with the format path's timeout handling. Review charts/latest/values.yaml and reproduce the fsck OOM/crashloop scenario if possible. Done means a supported handling of large-volume fsck prevents the unbreakable loop and gives operators a clear recovery path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, kubernetes, linux
- Domain
- devops, infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100