canonical / canonical/multipass

Daemon freeze during qemu crash on VM kernel panic

Open
#4,426 0 comments 0 reactions 0 assignees View on GitHub
backend (QEMU) needs triage research
Dominant language
C++
Stars
9.2k
Forks
829
Avg merge
3d 2h
Merged PRs (30d)
45

Description

**Describe the bug**
As part of attempt to reproduce #4385, I found out that when a VM has a kernel panic, the multipass daemon freezes. The daemon is able to recover in approx. 20 minutes. The leading theory is that the main thread launches a QMP via qemu to request data from the vm and that starts an I/O operation that cannot be interrupted by SIGINT, but can be interrupted by SIGKILL. From the logs it seems like qemu also crashes eventually. The daemon recovers gracefully but it takes a long time to do so, as illustrated in the logs.

**To Reproduce**
How, and what happened?
1. `multipass launch -n a`
2. `multipass exec a -- sudo sysctl vm.panic_on_oom=1`
3. Launch a process that causes an oom (a C program that mallocs the entire RAM).
4. The daemon should freeze the next time it tries to contact the vm.

**Expected behavior**
Communication with the panicked VM should be lost, qemu may crash, but the daemon should not completely freeze. Either by sending the QMP (if that is the problem) with a secondary thread, or by detecting the panic and isolating the VM from future communications to avoid the freeze. Another option (which does work if executed really fast manually), is to `multipass stop --force` the VM before it is polled via QMP. The problem of that type of solutions is detection.

**Logs**
```
Oct 09 14:55:06 HungryBox multipassd[224447]: QMP: {"timestamp": {"seconds": 1760014506, "microseconds": 502119}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[4]/rtc"}}
Oct 09 15:15:30 HungryBox multipassd[224447]: SSH session disconnected: [ssh proc] failed to open session channel: 'Socket error: No route to host'
Oct 09 15:15:30 HungryBox multipassd[224447]: Renewing cached SSH session
Oct 09 15:15:33 HungryBox multipassd[224447]: Forcing shutdown
Oct 09 15:15:33 HungryBox multipassd[224447]: Killing process
Oct 09 15:15:33 HungryBox multipassd[224447]: process state changed to NotRunning
Oct 09 15:15:33 HungryBox multipassd[224447]: process error occurred Crashed program: qemu-system-x86_64; error: Process crashed
Oct 09 15:15:33 HungryBox multipassd[224447]: Saved file `/root/.local/share/multipassd/multipassd-vm-instances.json` successfully in attempt #1
Oct 09 15:15:33 HungryBox multipassd[224447]: error: program: qemu-system-x86_64; error: Process crashed
Oct 09 15:15:33 HungryBox multipassd[224447]: Saved file `/root/.local/share/multipassd/multipassd-vm-instances.json` successfully in attempt #1
Oct 09 15:15:33 HungryBox multipassd[224447]: Applied AppArmor policy: multipass.qemu-img
Oct 09 15:15:33 HungryBox multipassd[224447]: [228446] started: qemu-img snapshot -l /root/.local/share/multipassd/vault/instances/a/ubuntu-24.04-server-cloudimg-amd64.img
Oct 09 15:15:33 HungryBox multipassd[224447]: fetch manifest periodically
```
Logs before 14:55 are uninformative to the freeze.

**Additional info**
- OS: Ubuntu 24.04.03
- CPU architecture or model: x86
- `multipass version` : 1.17.0-dev.509+gfdcfdcb0
- `multipass get local.driver` : qemu

**To test**
- Attempt to trace the point in the daemon where the freeze happens by running the daemon with gdb and sending SIGTRAP after the freeze and see if that signal is not ignored.
- Behavior under other backends (Windows and macOS).

**Possible solutions**
Actively query the process for OS panics, e.g. in qemu: [guest-panicked – guest has been panicked as a result of guest OS panic](https://qemu-project.gitlab.io/qemu/interop/qemu-qmp-ref.html#enum-QMP-run-state.RunState)

Contributor guide

Open the contributing guide

Research direction

Reproduce the freeze with `multipass launch`, `multipass exec`, and a VM memory-exhaustion process, then inspect the daemon with gdb and SIGTRAP as suggested. Read the QMP `guest-panicked` run-state documentation and compare behavior across backends. Done means a panicked VM can lose communication or stop without freezing the daemon.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.