abiosoft / abiosoft/colima

krunkit: secondary "disk:" volume mounts on first boot but breaks ("Unexpected disk at mount point.") on every subsequent restart

Aperta
#1,614 0 commenti 4 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
30.8k
Fork
613
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### Description

On a `krunkit` vmType profile, the secondary data disk (the one sized by `disk:` in `colima.yaml`, as opposed to `rootDisk:`) mounts correctly on the VM's **first boot**, but is silently left **unmounted** after any subsequent plain `colima stop` / `colima start` cycle — no config change, no delete/recreate needed. `provision.dependency/00000007` (`disk.sh`) fails on that second boot with `"Unexpected disk at mount point."`, and a separate root-disk resize step (`provision.system/00000006`) also starts failing at the same time with `resize2fs: Please run 'e2fsck -f ...' first`, pointed at the very disk that `disk.sh` is trying (and failing) to mount. Both failures make `cloud-final.service` exit with status 1 (`failed`).

Net effect: container/model data (`/var/lib/docker`, `/var/lib/containerd`, `/var/lib/ramalama`, etc.) silently falls back onto the small `rootDisk` (default 20GiB) instead of the large `disk:` volume after the first restart, so the VM runs out of space almost immediately when pulling large images/models (e.g. `colima model serve --runner ramalama <30B model>` fails with "no space left on device").

Reproduced twice independently: once on a long-lived profile (`krunkit`, delete + recreate did not help), and once from scratch on a brand-new profile created solely to isolate this (`krunkit-repro`) — full commands and output below. The equivalent `vz` profile (`default`, macOS Virtualization.framework) mounts its secondary disk correctly on every boot and has no such issue.

I checked `main` at HEAD (past the v0.10.3 tag): neither `environment/vm/lima/disk.sh` nor `environment/vm/lima/yaml.go` (which owns the `resize2fs`/`growpart` root-disk logic) has any commits since v0.10.3, so this is still present on current `main`, not just the released version.

Likely related to #1570/#1571 (fixed the "already mounted" detection in `disk.sh` using `findmnt`, present in my version) — but a different branch of the same script (the second `findmnt --target` guard, `"Unexpected disk at mount point."`) is the one failing here. Also very likely the underlying cause of #1583 ("krunkit vm data lost on restart"), which reports the same "works once, breaks on restart" pattern.

Separately, note for maintainers: while reproducing this on a fresh profile I also hit #1607 (`mountType must be virtiofs or reverse-sshfs, got 9p`) on plain `colima start -p --vm-type krunkit`. That one is **already fixed on `main`** (#1609, merged after v0.10.3) but not yet in a tagged release — mentioning it only so it isn't confused with this report; I worked around it with `--mount-type sshfs` to get to the actual repro below.

### Version

```
colima version 0.10.3
git commit: 00f6c297e92a82c04a4ab507db0a61435650d7e8

runtime: docker
arch: aarch64
client: v28.5.1
server: v29.5.2
limactl version 2.2.0
qemu-img version 11.0.3

krunkit 1.2.2
```

### Operating System

- [ ] macOS Intel <= 13 (Ventura)
- [ ] macOS Intel >= 14 (Sonoma)
- [x] Apple Silicon >= 14 (Sonoma) — macOS 26.6 (25G72)
- [ ] Linux

### Reproduction Steps (clean, isolated profile)

```
$ colima start -p krunkit-repro --vm-type krunkit --disk 40
...
level=error msg="Failed to set config" error="rpc error: code = Unknown desc = field `mountType` must be `virtiofs` or `reverse-sshfs` for krunkit driver, got `9p`"
```
(→ this is #1607, fixed on `main` via #1609 but not yet released; worked around below.)

```
$ colima delete -p krunkit-repro -y
$ colima start -p krunkit-repro --vm-type krunkit --mount-type sshfs --disk 40
... (starts fine)
```

**First boot — disk mounts correctly:**
```
$ colima ssh -p krunkit-repro -- lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 253:0 0 20G 0 disk
├─vda1 253:1 0 19G 0 part /
├─vda15 253:15 0 99M 0 part /boot/efi
└─vda16 259:0 0 923M 0 part /boot
vdb 253:16 0 19M 0 disk /mnt/lima-cidata
vdc 253:32 0 40G 0 disk
└─vdc1 253:33 0 40G 0 part /var/lib/ramalama
/var/lib/cni
/var/lib/rancher
/var/lib/containerd
/var/lib/docker
/mnt/lima-colima-krunkit-repro

$ colima ssh -p krunkit-repro -- systemctl is-active cloud-final.service
failed # (this boot's failure was an unrelated transient DNS-not-ready issue during
# apt-get install rsync in boot.Linux/30-install-packages.sh — disk itself
# mounted fine, included for completeness)
```

**Restart with zero config changes — disk comes back unmounted:**
```
$ colima stop -p krunkit-repro
$ colima start -p krunkit-repro
$ colima ssh -p krunkit-repro -- lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 253:0 0 20G 0 disk
├─vda1 253:1 0 19G 0 part /var/lib/ramalama
│ /var/lib/cni
│ /var/lib/rancher
│ /var/lib/containerd
│ /var/lib/docker
│ /
├─vda15 253:15 0 99M 0 part /boot/efi
└─vda16 259:0 0 923M 0 part /boot
vdb 253:16 0 19M 0 disk /mnt/lima-cidata
vdc 253:32 0 39.9G 0 disk
└─vdc1 253:33 0 39.9G 0 part # <- no mountpoint anymore

$ colima ssh -p krunkit-repro -- systemctl is-active cloud-final.service
failed
```

**Journal for that failing boot:**
```
$ colima ssh -p krunkit-repro -- sudo journalctl -b -u cloud-final.service -o cat
...
+ [ -d /mnt/lima-cidata/provision.dependency ]
+ CODE=0
+ /mnt/lima-cidata/provision.dependency/00000001
+ /mnt/lima-cidata/provision.dependency/00000007
/dev/vdb 20M 20M 0 100% /mnt/lima-cidata
Unexpected disk at mount point.
+ CODE=1
+ /mnt/lima-cidata/provision.dependency/00000008
...
+ [ 1 != 0 ]
+ exit 1
LIMA ...| WARNING: Failed to execute /mnt/lima-cidata/boot.Linux/30-install-packages.sh
...
LIMA ...| Executing /mnt/lima-cidata/provision.system/00000006
resize2fs 1.47.0 (5-Feb-2023)
Please run 'e2fsck -f /dev/disk/by-label/lima-colima-krun' first.
LIMA ...| Exiting with code 1
...
cloud-final.service: Main process exited, code=exited, status=1/FAILURE
cloud-final.service: Failed with result 'exit-code'.
Failed to start cloud-final.service - Cloud-init: Final Stage.
```

`blkid` confirms `/dev/disk/by-label/lima-colima-krun` unambiguously resolves to `/dev/vdc1` (the secondary disk; the root disk is separately labelled `cloudimg-rootfs`), so `provision.system/00000006`'s `resize2fs` step is being run against the secondary disk, which apparently didn't get cleanly unmounted at `colima stop` and now needs `e2fsck` before it can be touched — which then aborts the whole provisioning chain with `set -e`-style propagation, so `disk.sh`'s mount attempt right before it (`00000007`) never gets a clean run either.

```
$ colima ssh -p krunkit-repro -- sudo blkid
/dev/vdb: BLOCK_SIZE="2048" UUID="..." LABEL="cidata" TYPE="iso9660"
/dev/vdc1: LABEL="lima-colima-krun" UUID="..." BLOCK_SIZE="4096" TYPE="ext4"
/dev/vda15: LABEL_FATBOOT="UEFI" LABEL="UEFI" ...
/dev/vda1: LABEL="cloudimg-rootfs" UUID="..." BLOCK_SIZE="4096" TYPE="ext4" ...
/dev/vda16: LABEL="BOOT" UUID="..." ...
```

For comparison, the `default` profile (`vmType: vz`) on the same host mounts its equivalent disk fine on every boot:
```
$ colima ssh -p default -- df -h
...
/dev/vdb1 59G 36G 21G 64% /mnt/lima-colima
$ colima ssh -p default -- systemctl is-active cloud-final.service
active
```

### Expected behaviour

- `disk.sh` mounts the secondary disk on every boot on `krunkit`, not just the first.
- The root-disk resize step (`provision.system/00000006`) never targets the secondary disk.
- `cloud-final.service` completes successfully (`active`) after a normal `stop`/`start`, not `failed`.
- Container/model data stays backed by the large `disk:` volume rather than silently falling back to the small `rootDisk` after a restart.

### Workaround

Bumping `rootDisk` instead of relying on `disk:` works around the space problem (though `cloud-final.service` still ends up `failed` for the reasons above):
```
colima stop -p krunkit
colima start -p krunkit --root-disk 100
```
`vda1` grows in place via colima's own growpart/resize2fs and everything under `/var/lib` gets the extra space, independent of the broken second-disk mount.

### Additional context

- Confirmed present on current `main` (past v0.10.3): no commits to `environment/vm/lima/disk.sh` or `environment/vm/lima/yaml.go` since the v0.10.3 tag.
- Ruled out the fstrim/discard bug in #1549 (`provision.system/00000005`) as the cause here — manually re-running that script inside the VM exits `0` cleanly on this image.
- Ruled out ext4 16-char label truncation/collision between root and secondary disk as the cause — `blkid` shows distinct labels (`cloudimg-rootfs` vs `lima-colima-krun`).
- Happy to provide the full raw `journalctl -b -u cloud-final.service` output (trimmed above for brevity) or run further diagnostics.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.