apache / apache/cloudstack

VM deployment randomly fails on XCP-ng 8.3 due to stale template_spool_ref

Open
#14,081 1 comment 0 reactions 0 assignees View on GitHub
status:needs-reproducing
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

We are experiencing intermittent/random VM deployment failures in CloudStack 4.22.1 with XCP-ng 8.3.

During the investigation, we observed that the template copy/base volume associated with the deployment is no longer present in the primary storage. However, the corresponding database entry still exists in the template_spool_ref table.

When CloudStack subsequently attempts to deploy a VM from the affected template, it appears to retrieve the stale storage reference from the database and attempts to access a VDI that no longer exists.

The deployment then fails with an XCP-ng API error indicating that the VDI UUID is invalid.

### versions

Apache CloudStack: 4.22.1
Hypervisor: XCP-ng 8.3
Storage: Presetup [iSCSI]

### The steps to reproduce the bug

VM Launch is failing below is observed in logs:
```

2026-09-08 15:02:51,295 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].
2026-09-08 15:02:51,295 WARN [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].
```

### What to do about it?

Should the corresponding template_spool_ref entry be removed or marked appropriately when the template copy is deleted?

Contributor guide

Open the contributing guide

Research direction

Start by tracing how template_spool_ref entries are created and handled when a template copy or base volume is deleted, using the reported XCP-ng deployment failure as the reproduction signal. Determine the appropriate cleanup behavior and verify that deploying from an affected template no longer attempts to access a missing VDI.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.