Azure / Azure/local-csi-driver

fsck OOM-crashloop on csi-local-node after AKS node upgrade (10 TiB ext4, 600 Mi limit)

Open
#458 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
33
Forks
16
Avg merge
10h 17m
Merged PRs (30d)
28

Description

## What happened

After an AKS node-image upgrade, the `csi-local-node` DaemonSet's `driver` container is OOM-killed in a tight crashloop while staging a previously formatted local volume. The pod consuming the PVC stays in `ContainerCreating` indefinitely with `MountVolume.MountDevice failed ... context deadline exceeded`.

The volume is ~10 TiB of ext4 holding many small files (high inode count). Each pod restart re-runs `fsck` from the top, hits the 600 Mi limit again, and gets killed. We could not break the loop without intervention.

**Mitigation we used:** delete and recreate the PVC (data was cache and could be rebuilt). We don't believe this should be the only option.

## Why it happens

`internal/csi/mounter/mounter.go` embeds `k8s.io/mount-utils.SafeFormatAndMount`, whose `formatAndMountSensitive` flow calls `checkAndRepairFilesystem` -> `fsck -a` synchronously before mount on any already-formatted rw volume ([upstream code](https://github.com/kubernetes/mount-utils/blob/master/mount_linux.go#L530)). For ext4, `fsck -a` dispatches to `e2fsck -p` (preen mode). On a clean unmount this is effectively a no-op, but after an ungraceful detach during the node upgrade the filesystem state was dirty and `e2fsck` entered repair mode.

Peak `e2fsck` memory scales with the working set of inode bitmaps, the `icount` structures (pass 1), and `dx_dir` / orphan-list state (pass 2). For a 10 TiB ext4 with the default `bytes-per-inode = 16384` (~640M inodes, [`mke2fs.conf` defaults](https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/plain/misc/mke2fs.conf.in)) holding millions of small files, repair-mode RSS routinely exceeds the 600 Mi container limit set in `charts/latest/values.yaml`:

```yaml
driver:
limits:
memory: 600Mi
requests:
cpu: 10m
memory: 60Mi
```

The fact that e2fsprogs ships an e2fsck.conf(5) scratch_files option specifically to spill these structures to disk on memory-constrained systems is itself evidence that this working set can grow well past sub-gigabyte limits on large filesystems. Azure/AKS#4682 has an independent report of the same OOM on a 12,500 GiB azuredisk volume against the same 600 Mi node-DaemonSet limit.

Two compounding factors turn the OOM into an unbreakable crashloop:

1. No timeout on fsck. SafeFormatAndMount.checkAndRepairFilesystem calls mounter.Exec.Command("fsck", "-a", source).CombinedOutput() with no context.Context, no timer, and no semaphore - unlike the format path right next to it which does honour mounter.formatTimeout. Combined with the OOM kill, every restart re-enters fsck from scratch and the loop never terminates.
2. No skip-fsck escape hatch at the mount-utils layer or as a CSI driver flag / StorageClass option. Once you're in this state, the only way out is recreating the volume or running e2fsck out-of-band on the host.

Prior art

This pattern has been reported against multiple CSI drivers and is not yet fixed upstream:

- Azure/AKS#4682 - azuredisk-csi-node OOM-killed during fsck (12,500 GiB volume, 600 Mi limit; closed-as-stale in June 2025 without a code fix).
- Azure/AKS#4421 - earlier related thread on memory pressure in the same node DaemonSet.

Repro

1. Provision a local.csi.acstor.io PVC, ~10 TiB, ext4.
2. Fill it with many small files (push inode count into the hundreds of millions).
3. Trigger an ungraceful node detach (AKS node-image upgrade, or reboot -f on the node).
4. After the node comes back, observe the csi-local-node driver container OOMKilled with the last log line being Checking for issues with fsck on disk: /dev/....
5. Pod consuming the PVC stays in ContainerCreating; kubectl get events shows MountVolume.MountDevice failed ... DeadlineExceeded.

Expected behavior

Any one of:

- The node DaemonSet has enough memory headroom to complete e2fsck -p on realistic large volumes; or
- There is a documented, supported way to skip or bound the in-line fsck (per-StorageClass parameter, per-PV annotation, or driver flag); or
- The OOM does not produce an unbreakable crashloop - e.g. fsck timeout + the driver fails the NodeStageVolume RPC cleanly with a clear error message so the operator can intervene.

Environment

- Kubernetes version: 1.35.0
- local-csi-driver version: v0.2.10 (we have not retested on v0.2.14 though nothing has changed there)
- Volume size / fs type / approx file count: ~10 TiB / ext4 / millions of small files

Contributor guide

Open the contributing guide

Research direction

Start with internal/csi/mounter/mounter.go and the SafeFormatAndMount checkAndRepairFilesystem flow described in the report; compare it with the format path's timeout handling. Review charts/latest/values.yaml and reproduce the fsck OOM/crashloop scenario if possible. Done means a supported handling of large-volume fsck prevents the unbreakable loop and gives operators a clear recovery path.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, kubernetes, linux
Domain
devops, infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.