abiosoft / abiosoft/colima

krunkit: secondary "disk:" volume mounts on first boot but breaks ("Unexpected disk at mount point.") on every subsequent restart

Abierto
#1,614 0 comentarios 4 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Go
Estrellas
30.8k
Forks
613
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Description

On a `krunkit` vmType profile, the secondary data disk (the one sized by `disk:` in `colima.yaml`, as opposed to `rootDisk:`) mounts correctly on the VM's **first boot**, but is silently left **unmounted** after any subsequent plain `colima stop` / `colima start` cycle — no config change, no delete/recreate needed. `provision.dependency/00000007` (`disk.sh`) fails on that second boot with `"Unexpected disk at mount point."`, and a separate root-disk resize step (`provision.system/00000006`) also starts failing at the same time with `resize2fs: Please run 'e2fsck -f ...' first`, pointed at the very disk that `disk.sh` is trying (and failing) to mount. Both failures make `cloud-final.service` exit with status 1 (`failed`).

Net effect: container/model data (`/var/lib/docker`, `/var/lib/containerd`, `/var/lib/ramalama`, etc.) silently falls back onto the small `rootDisk` (default 20GiB) instead of the large `disk:` volume after the first restart, so the VM runs out of space almost immediately when pulling large images/models (e.g. `colima model serve --runner ramalama <30B model>` fails with "no space left on device").

Reproduced twice independently: once on a long-lived profile (`krunkit`, delete + recreate did not help), and once from scratch on a brand-new profile created solely to isolate this (`krunkit-repro`) — full commands and output below. The equivalent `vz` profile (`default`, macOS Virtualization.framework) mounts its secondary disk correctly on every boot and has no such issue.

I checked `main` at HEAD (past the v0.10.3 tag): neither `environment/vm/lima/disk.sh` nor `environment/vm/lima/yaml.go` (which owns the `resize2fs`/`growpart` root-disk logic) has any commits since v0.10.3, so this is still present on current `main`, not just the released version.

Likely related to #1570/#1571 (fixed the "already mounted" detection in `disk.sh` using `findmnt`, present in my version) — but a different branch of the same script (the second `findmnt --target` guard, `"Unexpected disk at mount point."`) is the one failing here. Also very likely the underlying cause of #1583 ("krunkit vm data lost on restart"), which reports the same "works once, breaks on restart" pattern.

Separately, note for maintainers: while reproducing this on a fresh profile I also hit #1607 (`mountType must be virtiofs or reverse-sshfs, got 9p`) on plain `colima start -p --vm-type krunkit`. That one is **already fixed on `main`** (#1609, merged after v0.10.3) but not yet in a tagged release — mentioning it only so it isn't confused with this report; I worked around it with `--mount-type sshfs` to get to the actual repro below.

### Version

```
colima version 0.10.3
git commit: 00f6c297e92a82c04a4ab507db0a61435650d7e8

runtime: docker
arch: aarch64
client: v28.5.1
server: v29.5.2
limactl version 2.2.0
qemu-img version 11.0.3

krunkit 1.2.2
```

### Operating System

- [ ] macOS Intel <= 13 (Ventura)
- [ ] macOS Intel >= 14 (Sonoma)
- [x] Apple Silicon >= 14 (Sonoma) — macOS 26.6 (25G72)
- [ ] Linux

### Reproduction Steps (clean, isolated profile)

```
$ colima start -p krunkit-repro --vm-type krunkit --disk 40
...
level=error msg="Failed to set config" error="rpc error: code = Unknown desc = field `mountType` must be `virtiofs` or `reverse-sshfs` for krunkit driver, got `9p`"
```
(→ this is #1607, fixed on `main` via #1609 but not yet released; worked around below.)

```
$ colima delete -p krunkit-repro -y
$ colima start -p krunkit-repro --vm-type krunkit --mount-type sshfs --disk 40
... (starts fine)
```

**First boot — disk mounts correctly:**
```
$ colima ssh -p krunkit-repro -- lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 253:0 0 20G 0 disk
├─vda1 253:1 0 19G 0 part /
├─vda15 253:15 0 99M 0 part /boot/efi
└─vda16 259:0 0 923M 0 part /boot
vdb 253:16 0 19M 0 disk /mnt/lima-cidata
vdc 253:32 0 40G 0 disk
└─vdc1 253:33 0 40G 0 part /var/lib/ramalama
/var/lib/cni
/var/lib/rancher
/var/lib/containerd
/var/lib/docker
/mnt/lima-colima-krunkit-repro

$ colima ssh -p krunkit-repro -- systemctl is-active cloud-final.service
failed # (this boot's failure was an unrelated transient DNS-not-ready issue during
# apt-get install rsync in boot.Linux/30-install-packages.sh — disk itself
# mounted fine, included for completeness)
```

**Restart with zero config changes — disk comes back unmounted:**
```
$ colima stop -p krunkit-repro
$ colima start -p krunkit-repro
$ colima ssh -p krunkit-repro -- lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 253:0 0 20G 0 disk
├─vda1 253:1 0 19G 0 part /var/lib/ramalama
│ /var/lib/cni
│ /var/lib/rancher
│ /var/lib/containerd
│ /var/lib/docker
│ /
├─vda15 253:15 0 99M 0 part /boot/efi
└─vda16 259:0 0 923M 0 part /boot
vdb 253:16 0 19M 0 disk /mnt/lima-cidata
vdc 253:32 0 39.9G 0 disk
└─vdc1 253:33 0 39.9G 0 part # <- no mountpoint anymore

$ colima ssh -p krunkit-repro -- systemctl is-active cloud-final.service
failed
```

**Journal for that failing boot:**
```
$ colima ssh -p krunkit-repro -- sudo journalctl -b -u cloud-final.service -o cat
...
+ [ -d /mnt/lima-cidata/provision.dependency ]
+ CODE=0
+ /mnt/lima-cidata/provision.dependency/00000001
+ /mnt/lima-cidata/provision.dependency/00000007
/dev/vdb 20M 20M 0 100% /mnt/lima-cidata
Unexpected disk at mount point.
+ CODE=1
+ /mnt/lima-cidata/provision.dependency/00000008
...
+ [ 1 != 0 ]
+ exit 1
LIMA ...| WARNING: Failed to execute /mnt/lima-cidata/boot.Linux/30-install-packages.sh
...
LIMA ...| Executing /mnt/lima-cidata/provision.system/00000006
resize2fs 1.47.0 (5-Feb-2023)
Please run 'e2fsck -f /dev/disk/by-label/lima-colima-krun' first.
LIMA ...| Exiting with code 1
...
cloud-final.service: Main process exited, code=exited, status=1/FAILURE
cloud-final.service: Failed with result 'exit-code'.
Failed to start cloud-final.service - Cloud-init: Final Stage.
```

`blkid` confirms `/dev/disk/by-label/lima-colima-krun` unambiguously resolves to `/dev/vdc1` (the secondary disk; the root disk is separately labelled `cloudimg-rootfs`), so `provision.system/00000006`'s `resize2fs` step is being run against the secondary disk, which apparently didn't get cleanly unmounted at `colima stop` and now needs `e2fsck` before it can be touched — which then aborts the whole provisioning chain with `set -e`-style propagation, so `disk.sh`'s mount attempt right before it (`00000007`) never gets a clean run either.

```
$ colima ssh -p krunkit-repro -- sudo blkid
/dev/vdb: BLOCK_SIZE="2048" UUID="..." LABEL="cidata" TYPE="iso9660"
/dev/vdc1: LABEL="lima-colima-krun" UUID="..." BLOCK_SIZE="4096" TYPE="ext4"
/dev/vda15: LABEL_FATBOOT="UEFI" LABEL="UEFI" ...
/dev/vda1: LABEL="cloudimg-rootfs" UUID="..." BLOCK_SIZE="4096" TYPE="ext4" ...
/dev/vda16: LABEL="BOOT" UUID="..." ...
```

For comparison, the `default` profile (`vmType: vz`) on the same host mounts its equivalent disk fine on every boot:
```
$ colima ssh -p default -- df -h
...
/dev/vdb1 59G 36G 21G 64% /mnt/lima-colima
$ colima ssh -p default -- systemctl is-active cloud-final.service
active
```

### Expected behaviour

- `disk.sh` mounts the secondary disk on every boot on `krunkit`, not just the first.
- The root-disk resize step (`provision.system/00000006`) never targets the secondary disk.
- `cloud-final.service` completes successfully (`active`) after a normal `stop`/`start`, not `failed`.
- Container/model data stays backed by the large `disk:` volume rather than silently falling back to the small `rootDisk` after a restart.

### Workaround

Bumping `rootDisk` instead of relying on `disk:` works around the space problem (though `cloud-final.service` still ends up `failed` for the reasons above):
```
colima stop -p krunkit
colima start -p krunkit --root-disk 100
```
`vda1` grows in place via colima's own growpart/resize2fs and everything under `/var/lib` gets the extra space, independent of the broken second-disk mount.

### Additional context

- Confirmed present on current `main` (past v0.10.3): no commits to `environment/vm/lima/disk.sh` or `environment/vm/lima/yaml.go` since the v0.10.3 tag.
- Ruled out the fstrim/discard bug in #1549 (`provision.system/00000005`) as the cause here — manually re-running that script inside the VM exits `0` cleanly on this image.
- Ruled out ext4 16-char label truncation/collision between root and secondary disk as the cause — `blkid` shows distinct labels (`cloudimg-rootfs` vs `lima-colima-krun`).
- Happy to provide the full raw `journalctl -b -u cloud-final.service` output (trimmed above for brevity) or run further diagnostics.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.