bottlerocket-os / bottlerocket-os/bottlerocket

XFS statfs free-space grossly wrong vs superblock on 6.1.148 → kubelet false disk pressure

Open
#4,652 1 comment 0 reactions 0 assignees View on GitHub
status/needs-triage type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 18h
Merged PRs (30d)
12

Description

**Image I'm using:**

Bottlerocket 1.47.0 (aws-k8s-1.32)
Kernel: 6.1.148 #1 SMP PREEMPT_DYNAMIC Mon Sep 8 23:27:00 UTC 2025 x86_64 GNU/Linux
Filesystem: XFS on /dev/nvme1n1p1 (300 GB volume)

**What I expected to happen:**

`df` / `statfs` should accurately report free/used space, consistent with the XFS superblock, so that kubelet does not incorrectly mark the node as having DiskPressure.

**What actually happened:**

`statfs` (and therefore `df` and kubelet) reports that ~250 GiB of the 300 GiB disk is used, while the XFS superblock reports only ~2.3 GiB used.
Kubelet evicts pods due to false DiskPressure.

**Examples:**

`statfs` (df / kubelet view):
```bash
findmnt -T /var/lib/kubelet
# TARGET SOURCE FSTYPE OPTIONS
# /var /dev/nvme1n1p1[/var] xfs rw,nosuid,nodev,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

stat -f -c 'blocks=%b bfree=%f bavail=%a bsize=%s' /var/lib/kubelet
# blocks=78604289 bfree=12384292 bavail=12384292 bsize=4096
# → ~252 GiB used

# XFS superblock (authoritative):

xfs_db -r -c "sb 0" -c "p dblocks fdblocks" /dev/nvme1n1p1
# dblocks = 78642688
# fdblocks = 78047871
# → ~2.3 GiB used

# All mountpoints (/local, /var, /opt, /mnt) are bind-mounts of the same device.

# No large deleted-open files exist:

# PID=1851 COMM=dbus-broker-lau SIZE=2097152 FILE=/memfd:dbus-broker-log (deleted)
```

**How to reproduce the problem:**

Run a bunch of pods and they will start getting evicted:

```bash
ctr -n k8s.io containers list | wc -l
# 92
ctr -n k8s.io images | wc -l
# 25
du -sh /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
# 13G /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
du -sh /var/lib/containerd/io.containerd.content.v1.content/blobs/
# 3.5G /var/lib/containerd/io.containerd.content.v1.content/blobs/
du -sh /run/containerd/
# 54G /run/containerd/
df -h /var/lib/docker/
# Filesystem Size Used Avail Use% Mounted on
# /dev/nvme1n1p1 300G 255G 46G 85% /var
```

One more symptom - kubelet can't free up disk space:

```
usage=85 highThreshold=75 amountToFree=45866138419 lowThreshold=70
"Failed to garbage collect required amount of images. Attempted to free 45867354931 bytes, but only found 0 bytes eligible to free."
```

I can cordon and discard the node, but if I can provide any more info to catch some bug, please let me know.

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reproducing the statfs, df, and xfs_db discrepancy on the stated Bottlerocket and kernel environment, then trace how XFS free-space reporting reaches kubelet. Done means identifying the reporting mismatch and defining a validated correction that prevents false DiskPressure.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, linux
Domain
devops, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.