apache / apache/cloudstack

VM snapshot merge on KVM can leave volume.path in DB pointing to a file that no longer exists (multi-disk VMs)

Open
#14,030 4 comments 0 reactions 0 assignees View on GitHub
component:kvm type:bug
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

Summary:
When deleting a VM snapshot on KVM for a VM with more than one disk, each disk's data gets folded back into its real file one at a time, but CloudStack only gets a single "succeeded or failed" answer for the whole set. If an earlier disk's fold finishes for real but a later disk's then fails or times out, the whole thing is reported as failed, so CloudStack never updates its record for the disk that actually finished. Its database is left pointing to a file that no longer exists, with nothing to catch or fix this later. The VM then fails to start with "Can't find volume:", and the only current fix is to manually correct the database to match the real file.

Steps to reproduce:
1. Create a VM with two or more disks on KVM.
2. Take a VM snapshot, then delete it while the VM has enough disk activity that the merge takes a while (or induce a timeout/communication failure partway through the multi-disk merge).
3. If one disk's merge completes on the host before another disk's merge fails/times out, the completed disk's volumes.path is left stale.
4. Attempt to start the VM. It fails looking for the old file.

Environment where this was observed: KVM, disk-only VM snapshots, VM with 2 disks (ROOT + DATA), primary storage on NFS.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the KVM snapshot-deletion path for multi-disk merges and how its aggregate result updates volume paths; the issue provides no file or test names. Reproduce a partial merge failure, then verify that completed disks retain valid database paths and the VM can start without manual database repair.

Written by the indexing model from the issue text.

Assessment

Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.