vllm-project / vllm-project/aibrix

MountVolume.SetUp failed for volume "kube-api-access-wfmh2" no space left on device

Open
#694 2 comments 0 reactions 0 assignees View on GitHub
area/kv-cache kind/bug priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🐛 Describe the bug

![Image](https://github.com/user-attachments/assets/9a65f1b0-e39b-45ae-bd4d-c8762a4e3cbc)

```
Warning FailedMount 25s (x19 over 37m) kubelet (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-wfmh2" : write /var/lib/kubelet/pods/26af5e70-f9b9-4c12-a9de-dcf2d4e93689/volumes/kubernetes.io~projected/kube-api-access-wfmh2/..2025_02_18_00_11_37.99831085/namespace: no space left on device
```

```
Warning FailedMount 2m1s (x11 over 22m) kubelet (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-tzrpb" : write /var/lib/kubelet/pods/a951d40b-518a-4b29-a3c9-36ef96b4f09c/volumes/kubernetes.io~projected/kube-api-access-tzrpb/..2025_02_17_23_18_11.2933250311/ca.crt: no space left on device

Filesystem Size Used Avail Use% Mounted on
/dev/root 2.0G 1.2G 775M 61% /
devtmpfs 24G 0 24G 0% /dev
tmpfs 24G 0 24G 0% /dev/shm
tmpfs 9.5G 6.2M 9.5G 1% /run
tmpfs 24G 248K 24G 1% /etc/machine-id
tmpfs 256K 0 256K 0% /mnt/disks
efivarfs 56K 24K 27K 48% /sys/firmware/efi/efivars
tmpfs 24G 0 24G 0% /tmp
overlayfs 24G 248K 24G 1% /etc
/dev/nvme0n1p8 11M 24K 11M 1% /usr/share/oem
/dev/nvme0n1p1 95G 57G 38G 60% /mnt/stateful_partition
tmpfs 2.0M 224K 1.8M 11% /var/lib/cloud
none 24G 38M 24G 1% /dev/kube_shm
tmpfs 24G 12K 24G 1% /var/lib/kubelet/pki
```

If has enough room but failed

### Steps to Reproduce

latest version

```
kubectl apply -f samples/kvcache/kvcache.yaml
kubectl apply -f samples/kvcache/deployment.yaml
```

### Expected behavior

it should not through any errors

### Environment

nightly & GCP

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.