VM deployment randomly fails on XCP-ng 8.3 due to stale template_spool_ref
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 6d 19h
- Merged PRs (30d)
- 32
Description
### problem
We are experiencing intermittent/random VM deployment failures in CloudStack 4.22.1 with XCP-ng 8.3.
During the investigation, we observed that the template copy/base volume associated with the deployment is no longer present in the primary storage. However, the corresponding database entry still exists in the template_spool_ref table.
When CloudStack subsequently attempts to deploy a VM from the affected template, it appears to retrieve the stale storage reference from the database and attempts to access a VDI that no longer exists.
The deployment then fails with an XCP-ng API error indicating that the VDI UUID is invalid.
### versions
Apache CloudStack: 4.22.1
Hypervisor: XCP-ng 8.3
Storage: Presetup [iSCSI]
### The steps to reproduce the bug
VM Launch is failing below is observed in logs:
```
2026-09-08 15:02:51,295 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].
2026-09-08 15:02:51,295 WARN [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].
```
### What to do about it?
Should the corresponding template_spool_ref entry be removed or marked appropriately when the template copy is deleted?
Contributor guide
Research direction
Start by tracing how template_spool_ref entries are created and handled when a template copy or base volume is deleted, using the reported XCP-ng deployment failure as the reproduction signal. Determine the appropriate cleanup behavior and verify that deploying from an affected template no longer attempts to access a missing VDI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100