canonical / canonical/pycloudlib
Azure VM Quota Detection and Retry Logic
- Dominant language
- Python
- Stars
- 22
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Most of all resources in Azure are limited by a set quota per subscription (and per region). Exceeding the quota during a deployment will result in unrecoverable failure, as the Azure SDK does not implement retrying creation in case of certain failure modes. `pycloudlib` should implement quota detection and retry logic on provisioning failures due to quota at the `Azure._create_virtual_machine` layer for the following reasons:
- Retry logic *can* be implemented on top of `pycloudlib` by its users, but this involves the complete deletion and redeployment of all pre-requisite resources. `pycloudlib` abstracts the individual deployment steps, and creating the virtual machine itself is the final resource creation step. Should this stage fail, the user would be required to start from scratch as `pycloudlib` does not expose incremental resource deployments - the exception being if no network configuration is passed to the `launch` method (but this is an implementation detail that most users are not even aware of).
- While retry logic alone is sufficient as a technical requirement, attempting to provision VMs with no available quota is wasteful. `pycloudlib` should first check if the deployment is possible. However, there will always exist a race condition between the quota check and the deployment itself, which necessitates the retry logic on provisioning failure.
## Failure Example
```
16:23:54 pycloudlib.errors.PycloudlibError: Virtual machine creation error: OperationNotAllowed
16:23:54 Operation could not be completed as it results in exceeding approved standardLSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 24, Current Usage: 24, Additional Required: 8, (Minimum) New Limit Required: 32. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%229f899844-7a30-4d7c-bfbe-3d8e69851400%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22westeurope%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardLSv3Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:32,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardLSv3Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests`
```
## Proposed Enhancements
1. Implement retry logic on `Azure._create_virtual_machine`.
2. (Optional) Implement quota detection to mitigate excessive retries. Marked as optional as it will always be susceptible to race conditions.
Retry on provisioning failure can potentially be extended to all Azure resource deployments (as nearly all resources in Azure consume quota), but VM quota is a common bottleneck and should be prioritized.
Contributor guide
Assessment
This issue has not been assessed yet.