canonical / canonical/pycloudlib

Azure VM Quota Detection and Retry Logic

Open
#359 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22
Forks
42
PR merge metrics
No merged PRs in 30d

Description

## Summary

Most of all resources in Azure are limited by a set quota per subscription (and per region). Exceeding the quota during a deployment will result in unrecoverable failure, as the Azure SDK does not implement retrying creation in case of certain failure modes. `pycloudlib` should implement quota detection and retry logic on provisioning failures due to quota at the `Azure._create_virtual_machine` layer for the following reasons:

- Retry logic *can* be implemented on top of `pycloudlib` by its users, but this involves the complete deletion and redeployment of all pre-requisite resources. `pycloudlib` abstracts the individual deployment steps, and creating the virtual machine itself is the final resource creation step. Should this stage fail, the user would be required to start from scratch as `pycloudlib` does not expose incremental resource deployments - the exception being if no network configuration is passed to the `launch` method (but this is an implementation detail that most users are not even aware of).
- While retry logic alone is sufficient as a technical requirement, attempting to provision VMs with no available quota is wasteful. `pycloudlib` should first check if the deployment is possible. However, there will always exist a race condition between the quota check and the deployment itself, which necessitates the retry logic on provisioning failure.

## Failure Example

```
16:23:54 pycloudlib.errors.PycloudlibError: Virtual machine creation error: OperationNotAllowed
16:23:54 Operation could not be completed as it results in exceeding approved standardLSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 24, Current Usage: 24, Additional Required: 8, (Minimum) New Limit Required: 32. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%229f899844-7a30-4d7c-bfbe-3d8e69851400%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22westeurope%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardLSv3Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:32,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardLSv3Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests`
```

## Proposed Enhancements

1. Implement retry logic on `Azure._create_virtual_machine`.
2. (Optional) Implement quota detection to mitigate excessive retries. Marked as optional as it will always be susceptible to race conditions.

Retry on provisioning failure can potentially be extended to all Azure resource deployments (as nearly all resources in Azure consume quota), but VM quota is a common bottleneck and should be prioritized.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.