Azure / Azure/azure-cli

`az storage blob copy start` Should not require to query the source storage account key

Open
#28,167 3 comments 0 reactions 1 assignee Assigned to @calvinhzy View on GitHub
act-codegen-extensibility-squad act-platform-engineering-squad Auto-Assign Auto-Resolve Azure CLI Team customer-reported question Storage Upgrade
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

When performing an `az storage blob copy start` command to copy from one storage account to another with `--auth-mode login` & without `--source-account-key` the cli errors out unless the user has the capability to read the source storage account key.

This is because the cli uses the UploadFromURL api and creates a SAS url for the source blob. To do so it requires the storage account which if not provided queries it from azure directly.

### Related command

```shell
az storage blob copy start \
--auth-mode login \
--source-account-name "${SOURCE_ACCOUNT_NAME}" \
--source-container "${SOURCE_CONTAINER_NAME}" \
--source-blob "${file}" \
--account-name "${DESTINATION_ACCOUNT_NAME}" \
--destination-container "${DESTINATION_CONATINER_NAME}" \
--destination-blob "${file}"
```

### Errors

```
Source storage account "${SOURCE_ACCOUNT_NAME}" not found
```

### Issue script & Debug output

```
cli.knack.cli: Command arguments: ['storage', 'blob', 'copy', 'start', '--source-account-name', 'REDACTED', '--account-name', 'REDACTED', '--source-blob', 'silly', '--source-container', 'silly', '--destination-container', 'silly', '--destination-blob', 'silly', '--auth-mode', 'login', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [, , ]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: storage 0.049 57 269
cli.azure.cli.core: Total (1) 0.049 57 269
cli.azure.cli.core: Loaded 57 groups, 269 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : storage blob copy start
cli.azure.cli.core: Command table: storage blob copy start
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate []
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-01-10.19-15-46.storage_blob_copy_start.26.log'.
az_command_data_logger: command args: storage blob copy start --source-account-name {} --account-name {} --source-blob {} --source-container {} --destination-container {} --destination-blob {} --auth-mode {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x7fa60c79ab00>]
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 641, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 641, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'ListKeyExpand'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 641, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'CorsRuleAllowedMethodsItem'

cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x7fa60c550a60>, .add_cache_arguments at 0x7fa60c550b80>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [, , .parse_ids_arguments at 0x7fa60c550af0>]
cli.azure.cli.core.auth.persistence: build_persistence: location='/root/.azure/service_principal_entries.json', encrypt=False
cli.azure.cli.core.auth.persistence: build_persistence: location='/root/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /root/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
msal.application: Region to be used: None
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=StorageManagementClient
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
msal.application: Region to be used: None
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: d14f5a8b-dd7b-4d90-8a30-22febb9650f3
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/49523a5b-6879-448c-b2ab-ab8d5f2e9821/providers/Microsoft.Storage/storageAccounts?api-version=2022-09-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'a85addf6-afec-11ee-9f09-ee34ae0ea202'
cli.azure.cli.core.sdk.policies: 'CommandName': 'storage blob copy start'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--source-account-name --account-name --source-blob --source-container --destination-container --destination-blob --auth-mode --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.48.1 (DOCKER) azsdk-python-azure-mgmt-storage/21.0.0 Python/3.10.11 (Linux-5.15.0-1051-azure-x86_64-with)'
cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/49523a5b-6879-448c-b2ab-ab8d5f2e9821/providers/Microsoft.Storage/storageAccounts?api-version=2022-09-01 HTTP/1.1" 200 1497
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '1fdd5961-093e-4577-925f-cdf868dbff36'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '1fdd5961-093e-4577-925f-cdf868dbff36'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTEUROPE:20240110T191548Z:1fdd5961-093e-4577-925f-cdf868dbff36'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Date': 'Wed, 10 Jan 2024 19:15:47 GMT'
cli.azure.cli.core.sdk.policies: 'Content-Length': '1497'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"value":[{"sku":{"name":"Standard_ZRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/49523a5b-6879-448c-b2ab-ab8d5f2e9821/resourceGroups/storage/providers/Microsoft.Storage/storageAccounts/REDACTED","name":"REDACTED","type":"Microsoft.Storage/storageAccounts","location":"westeurope","tags":{},"properties":{"keyCreationTime":{"key1":null,"key2":null},"allowCrossTenantReplication":true,"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":true,"allowSharedKeyAccess":true,"isHnsEnabled":false,"networkAcls":{"resourceAccessRules":[],"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2020-02-06T10:35:41.8408699Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2020-02-06T10:35:41.8408699Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2020-02-06T10:35:41.7627044Z","primaryEndpoints":{"dfs":"https://REDACTED.dfs.core.windows.net/","web":"https://REDACTED.z6.web.core.windows.net/","blob":"https://REDACTED.blob.core.windows.net/","queue":"https://REDACTED.queue.core.windows.net/","table":"https://REDACTED.table.core.windows.net/","file":"https://REDACTED.file.core.windows.net/"},"primaryLocation":"westeurope","statusOfPrimary":"available"}}]}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 509, in validate_source_url
source_account_key = _query_account_key(cmd.cli_ctx, source_account_name)
File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 40, in _query_account_key
rg, scf = _query_account_rg(cli_ctx, account_name)
File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 58, in _query_account_rg
raise ValueError("Storage account '{}' not found.".format(account_name))
ValueError: Storage account 'REDACTED' not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 651, in execute
self._validation(expanded_arg)
File "/usr/local/lib/python3.10/site-packages/knack/invocation.py", line 113, in _validation
self._validate_arg_level(parsed_ns)
File "/usr/local/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 859, in _validate_arg_level
validator(**self._build_kwargs(validator, ns))
File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 511, in validate_source_url
raise RequiredArgumentMissingError('Source storage account {} not found.'.format(source_account_name))
azure.cli.core.azclierror.RequiredArgumentMissingError: Source storage account REDACTED not found.

cli.azure.cli.core.azclierror: Source storage account REDACTED not found.
az_command_data_logger: Source storage account REDACTED not found.
cli.knack.cli: Event: Cli.PostExecute []
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 1.600 seconds (init: 0.228, invoke: 1.372)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3623 in cache
telemetry.check: Negative: The /root/.azure/telemetry.txt was modified at 2024-01-10 19:14:06.337305, which in less than 600.000000 s
```

### Expected behavior

My expectation is that during the validation in `src/azure-cli/azure/cli/command_modules/storage/_validators.py:507` the codes doesn't ask for the storage account key to create a SAS, but instead detect if the user is able to create a [User delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas) using the existing login credentials and use that instead.

Allowing the command to succeed

### Environment Summary

azure-cli 2.48.1 *

core 2.48.1 *
telemetry 1.0.8 *

Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0

Python location '/usr/local/bin/python'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.10.11 (main, Apr 5 2023, 23:58:40) [GCC 12.2.1 20220924]

Legal docs and information: aka.ms/AzureCliLegal

You have 3 update(s) available. Consider updating your CLI installation with 'az upgrade'

### Additional context

This is the real reason for https://github.com/Azure/azure-cli/issues/9003

One expect that by granting a UserManagedIdentity (in my particular case) the `Storage Blob Data Reader` would suffice to copy data from that storage. The problem is that this command secretly requires the secret key (the equivalent of root mode) to be able to create SASs.

With this change the aforementioned role would suffice again since it includes the ability to create user delegation SASs. This is specially important from a security perspective since I only want to give read access to my Identity and the secret key would expose me to a risk of privilege escalation very easily.

I have been trying to edit the code myself but I am not very familiar with python and the way the [official documentation](https://learn.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blobserviceclient?view=azure-python#azure-storage-blob-blobserviceclient-get-user-delegation-key) makes use of the SDK is very different from what this code does.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.