Azure / Azure/azure-cli

az vm list-skus is very slow and sometimes fails

Open
#31,592 9 comments 8 reactions 1 assignee Assigned to @yanzhudd View on GitHub
act-observability-squad Auto-Assign Auto-Resolve Compute question Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

A command that is needed in automation is 'az vm list-skus' as we need to get specifics about a VM SKU before trying to create it.

However, the command, even when given very exacting information, is amazingly slow.

For example:

`az vm list-skus --location westus2 --resource-type virtualMachines --size Standard_D8plds_v5`

Takes on the order of 2 minutes+ when executed to return that one VM SKU's data. In some environments, this never actually succeeds (secure access workstations running Windows, for example)

The whole json for *all* sizes is only 77k with gzip compression and expended is 4.5meg. I can do jq queries over that data nearly instantly. (Unfortunately, getting the data is very, very slow) -- Looking at the debug output, it seems that there is even more data provided (122 megs) and then the filtering happens client side but that filtering is really, really slow.

This makes automation commands that provide for VM creation and provisioning really slow or they need to embed the information which is also not very good.

I have tried it with and without the `--resource-type` option to help the system filter better but it did not change the timing.

For example, I just ran this:
```bash
time az vm list-skus --location westus2 --resource-type virtualMachines --size Standard_D8plds_v5 >/dev/null

real 2m22.999s
user 2m3.916s
sys 0m1.653s
```

### Related command

The prior av vm list-sizes was also slow and is deprecated. It did not have a --size filter as users had to use the --query feature of Azure CLI to filter the output (which is just fine but maybe not as discoverable for those who are new to Azure CLI and/or json/jsonpath queries.

### Errors

In failure cases, it just sometimes hangs forever.

I have also seen it output this error (sometimes multiple times) and then either hang or eventually crash:

```
Traceback (most recent call last):
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\asyncio\windows_events.py", line 445, in select
RuntimeError: <_overlapped.Overlapped object at 0x0000017218925FB0> still has pending operation at deallocation, the process may crash
```

### Issue script & Debug output

`az vm list-skus --location westus2 --resource-type virtualMachines --size Standard_D8plds_v5 --debug`

The output is far too large to show here.

However, here is the output up to the point of the big data dump:
```
DEBUG: cli.knack.cli: Command arguments: ['vm', 'list-skus', '--location', 'westus2', '--resource-type', 'virtualMachines', '--size', 'Standard_D8plds_v5', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [, , ]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'vm': ['azure.cli.command_modules.vm']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: vm 0.375 59 228
DEBUG: cli.azure.cli.core: Total (1) 0.375 59 228
DEBUG: cli.azure.cli.core: Loaded 59 groups, 228 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command : vm list-skus
DEBUG: cli.azure.cli.core: Command table: vm list
DEBUG: cli.azure.cli.core: remaining : skus
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate []
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/msinz/.azure/commands/2025-06-03.15-10-08.vm_list-skus.1660582.log'.
INFO: az_command_data_logger: command args: vm list-skus --location {} --resource-type {} --size {} --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x7f3f7765c540>]
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.12/site-packages/azure/cli/core/profiles/_shared.py", line 471, in _get_attr
op = import_module(full_mod_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1310, in _find_and_load_unlocked
File "", line 488, in _call_with_frames_removed
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.mgmt.compute.v2024_03_03'

DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x7f3f7765ed40>, .add_cache_arguments at 0x7f3f7765ee80>, .update_breaking_change_info at 0x7f3f7765ef20>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [, , .parse_ids_arguments at 0x7f3f7765ede0>]
DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
DEBUG: cli.azure.cli.core.auth.msal_credentials: ManagedIdentityCredential.acquire_token: scopes=['https://management.core.windows.net//.default'], kwargs={}
DEBUG: msal.managed_identity: Obtaining token via managed identity on Azure VM
DEBUG: urllib3.connectionpool: Starting new HTTP connection (1): 169.254.169.254:80
DEBUG: urllib3.connectionpool: http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.core.windows.net%2F&msi_res_id=%2Fsubscriptions%2F******** HTTP/1.1" 200 2376
DEBUG: msal.token_cache: event={
"client_id": "/subscriptions/********",
"data": {},
"params": {},
"response": {
"access_token": "********",
"expires_in": 83894,
"refresh_in": 41947,
"resource": "https://management.core.windows.net/",
"token_type": "Bearer"
},
"scope": [
"https://management.core.windows.net/"
],
"token_endpoint": "https://********"
}
DEBUG: cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/********/providers/Microsoft.Compute/skus?api-version=2019-04-01'
DEBUG: cli.azure.cli.core.sdk.policies: Request method: 'GET'
DEBUG: cli.azure.cli.core.sdk.policies: Request headers:
DEBUG: cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'd60a353a-408c-11f0-b457-f7c6ba39c517'
DEBUG: cli.azure.cli.core.sdk.policies: 'CommandName': 'vm list-skus'
DEBUG: cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--location --resource-type --size --debug'
DEBUG: cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.73.0 (DEB) azsdk-python-core/1.31.0 Python/3.12.8 (Linux-5.15.0-1074-azure-x86_64-with-glibc2.35)'
DEBUG: cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
DEBUG: cli.azure.cli.core.sdk.policies: Request body:
DEBUG: cli.azure.cli.core.sdk.policies: This request has no body
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/********/providers/Microsoft.Compute/skus?api-version=2019-04-01 HTTP/1.1" 200 128930075
DEBUG: cli.azure.cli.core.sdk.policies: Response status: 200
DEBUG: cli.azure.cli.core.sdk.policies: Response headers:
DEBUG: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
DEBUG: cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Length': '128930075'
DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
DEBUG: cli.azure.cli.core.sdk.policies: 'Expires': '-1'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '1099'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-global-reads': '16499'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '16dba9dd-8966-460c-988d-ea180de34047'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '16dba9dd-8966-460c-988d-ea180de34047'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'EASTUS:20250603T151011Z:16dba9dd-8966-460c-988d-ea180de34047'
DEBUG: cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: 756939C06DF345D8AF0C87D4422DB442 Ref B: MWH011020808023 Ref C: 2025-06-03T15:10:08Z'
DEBUG: cli.azure.cli.core.sdk.policies: 'Date': 'Tue, 03 Jun 2025 15:10:16 GMT'
DEBUG: cli.azure.cli.core.sdk.policies: Response content:
```
... then after the big content dump ...
```
DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [, ]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
DEBUG: cli.knack.cli: Event: Cli.PostExecute []
INFO: az_command_data_logger: exit code: 0
INFO: cli.__main__: Command ran in 177.010 seconds (init: 0.201, invoke: 176.809)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3951 in cache file under /home/msinz/.azure/telemetry/20250603151304811
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.12/site-packages/azure/cli/telemetry/__init__.py /home/msinz/.azure /home/msinz
/.azure/telemetry/20250603151304811"
INFO: telemetry.process: Return from creating process 1661287
INFO: telemetry.main: Finish creating telemetry upload process.
```
... and now the SKU's data (note, not from the same run as I had lost the end bit and had to rerun...)
```
cli.knack.cli: Event: Cli.SuccessfulExecute []
cli.knack.cli: Event: Cli.PostExecute []
az_command_data_logger: exit code: 0
cli.__main__: Command ran in 152.182 seconds (init: 0.171, invoke: 152.011)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3951 in cache file under /home/msinz/.azure/telemetry/20250603152451814
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.12/site-packages/azure/cli/telemetry/__init__.py /home/msinz/.azure /home/msinz/.azure/telemetry/20250603152451814"
telemetry.process: Return from creating process 1664358
telemetry.main: Finish creating telemetry upload process.
```

### Expected behavior

A relatively quick return of the data for the VM (with `--query` operation on it or not)

This would allow for actually validating the SKU selected by the user and the parameters needed to construct it with the correct OS variations (CPU architectures, GPU support, accelerated networking, etc

### Environment Summary

I have different installations:

On Linux:
```
$ az --version
azure-cli 2.73.0 *

core 2.73.0 *
telemetry 1.1.0

Dependencies:
msal 1.32.3
azure-mgmt-resource 23.3.0

Python location '/opt/az/bin/python3'
Config directory '/home/msinz/.azure'
Extensions directory '/home/msinz/.azure/cliextensions'

Python (Linux) 3.12.8 (main, May 13 2025, 11:04:48) [GCC 11.4.0]
```

On secure access Windows machine:
```
azure-cli 2.55.0

core 2.55.0
telemetry 1.1.0

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\msinz.AME\.azure\cliextensions'

Python (Windows) 3.11.7 (remotes/origin/3.11.7/msft-spython:be2aebf208, Dec 11 2023, 22:59:37) [MSC v.1937 64 bit (AMD64)]
```

I have seen the crashes happen much more often on the Windows machine than Linux but the performance is equally as bad either place.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.