Azure / Azure/azure-cli

CLI should use private FQDN for Keyvaults if available

Open
#31,406 8 comments 0 reactions 1 assignee Claimed by @evelyn-ys View on GitHub
act-identity-squad Auto-Assign Auto-Resolve Azure CLI Team customer-reported KeyVault needs-team-triage Possible-Solution Python SDK - data plane question Similar-Issue
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

When configuring a Keyvault with private endpoint and no public access, I found that it never tries to use the private FQDN, instead always using the public FQDN.

Command: `az keyvault secret list --vault-name my-vault -o table`

Note, the vault name "my-vault" is not a real one I am using, simply changed the real one to avoid sharing the real hostname publicly.

Result:
```
(Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
```

In my hosts file, I added the privatelink FQDN `my-vault.privatelink.vaultcore.azure.net` pointing to the vault's private IP.

I expected for azure cli to try this FQDN, but instead I get the above error, indicating it is only trying the public FQDN `my-vault.vault.azure.net`

After adding a hosts file entry for `my-vault.vault.azure.net` pointing to the private IP, it works properly and I can list the secrets without issue.

This seems to be different from how the CLI interacts with our private Kubernetes cluster API server.

When I add either private or public FQDN to my /etc/hosts file specifically pointing at an unused private IP, or add both of them pointing to an unused private IP, then run `az aks command invoke -g my-resource-group -n my-cluster -c ls` I am able to get successful output from the command invocation which confirms that the cluster was reached despite me using an invalid API server IP address in my hosts file.

### Related command

`az keyvault secret list`

`az aks command invoke`

### Errors

(Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}

### Issue script & Debug output

(NOTE - Hostname has been redacted)

cli.knack.cli: Command arguments: ['keyvault', 'secret', 'list', '--vault-name', 'my-vault', '-o', 'table', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [, , ]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'keyvault': ['azure.cli.command_modules.keyvault']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: keyvault 0.004 20 113
cli.azure.cli.core: Total (1) 0.004 20 113
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: Loaded 20 groups, 113 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : keyvault secret list
cli.azure.cli.core: Command table: keyvault secret list
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate []
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/tom.spear/.azure/commands/2025-05-05.12-24-24.keyvault_secret_list.88926.log'.
az_command_data_logger: command args: keyvault secret list --vault-name {} -o {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x103c2add0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x103c2ae60>, .add_cache_arguments at 0x103c2af80>, .update_breaking_change_info at 0x103c2b010>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [, , .parse_ids_arguments at 0x103c2aef0>]
cli.azure.cli.core.auth.persistence: build_persistence: location='/Users/tom.spear/.azure/service_principal_entries.json', encrypt=False
cli.azure.cli.core.auth.persistence: build_persistence: location='/Users/tom.spear/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /Users/tom.spear/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8
msal.authority: openid_config("https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core._debug: Using CA bundle file at '/opt/homebrew/etc/ca-certificates/cert.pem'.
cli.azure.cli.core.sdk.policies: Request URL: 'https://my-vault.vault.azure.net/secrets?api-version=7.4'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'CommandName': 'keyvault secret list'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--vault-name -o --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.70.0 azsdk-python-core/1.32.0 Python/3.10.11 (macOS-15.4.1-arm64-arm-64bit)'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): my-vault.vault.azure.net:443
urllib3.connectionpool: https://my-vault.vault.azure.net:443 "GET /secrets?api-version=7.4 HTTP/1.1" 401 97
cli.azure.cli.core.sdk.policies: Response status: 401
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-region': 'eastus'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '56cc2881-3be4-4e08-8826-9be4d32fc563'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-service-version': '1.9.2336.1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-network-info': 'conn_type=Ipv4;addr=170.85.98.186;act_addr_fam=InterNetwork;'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains'
cli.azure.cli.core.sdk.policies: 'WWW-Authenticate': 'Bearer authorization="https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8", resource="https://vault.azure.net"'
cli.azure.cli.core.sdk.policies: 'Date': 'Mon, 05 May 2025 17:24:24 GMT'
cli.azure.cli.core.sdk.policies: 'Content-Length': '97'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"error":{"code":"Unauthorized","message":"AKV10000: Request is missing a Bearer or PoP token."}}
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://vault.azure.net/.default',), kwargs={'tenant_id': '92867854-71a8-42f2-898a-6953be9825e8'}
cli.azure.cli.core.auth.msal_credentials: ServicePrincipalCredential.get_token: scopes=('https://vault.azure.net/.default',), kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 4483d1c7-7f65-42b8-b80c-b69e0ca6c50b
cli.azure.cli.core.sdk.policies: Request URL: 'https://my-vault.vault.azure.net/secrets?api-version=7.4'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'CommandName': 'keyvault secret list'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--vault-name -o --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.70.0 azsdk-python-core/1.32.0 Python/3.10.11 (macOS-15.4.1-arm64-arm-64bit)'
cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: https://my-vault.vault.azure.net:443 "GET /secrets?api-version=7.4 HTTP/1.1" 403 297
cli.azure.cli.core.sdk.policies: Response status: 403
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-region': 'eastus'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '03fe41c2-b56a-4bfa-981b-62d3b71f1b01'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-service-version': '1.9.2336.1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-network-info': 'conn_type=Ipv4;addr=170.85.98.186;act_addr_fam=InterNetwork;'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains'
cli.azure.cli.core.sdk.policies: 'Date': 'Mon, 05 May 2025 17:24:24 GMT'
cli.azure.cli.core.sdk.policies: 'Content-Length': '297'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"error":{"code":"Forbidden","message":"Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'. \r\nVault: my-vault;location=eastus","innererror":{"code":"ForbiddenByConnection"}}}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 113, in keyvault_command_handler
return _encode_hex(transform_result(result, **{**command_args, 'kv_transform': True}))
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_transformers.py", line 12, in _multi_transformers
output = t(output, **command_args)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_transformers.py", line 29, in filter_out_managed_resources
return [_ for _ in output if not getattr(_, 'managed')] if output else output
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_transformers.py", line 29, in
return [_ for _ in output if not getattr(_, 'managed')] if output else output
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/core/paging.py", line 123, in __next__
return next(self._page_iterator)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/core/paging.py", line 75, in __next__
self._response = self._get_next(self.continuation_token)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/keyvault/secrets/_generated/v7_4/operations/_key_vault_client_operations.py", line 869, in get_next
raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
raise ex
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 734, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
result = cmd_copy(params)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
return self.handler(*args, **kwargs)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 135, in keyvault_command_handler
return keyvault_exception_handler(ex)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 49, in keyvault_exception_handler
raise CLIError(ex)
knack.util.CLIError: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}

cli.azure.cli.core.azclierror: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
az_command_data_logger: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
cli.knack.cli: Event: Cli.PostExecute []
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 0.616 seconds (init: 0.080, invoke: 0.535)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4412 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/bin/python3 /Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/telemetry/__init__.py /Users/tom.spear/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

### Expected behavior

CLI connects to keyvault using the private FQDN ending with `privatelink.vaultcore.azure.net` and shows the list of secrets.

### Environment Summary

azure-cli 2.70.0 *

core 2.70.0 *
telemetry 1.1.0

Extensions:
aks-preview 13.0.0b2

Dependencies:
msal 1.31.2b1
azure-mgmt-resource 23.1.1

Python location '/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/bin/python3'
Config directory '/Users/tom.spear/.azure'
Extensions directory '/Users/tom.spear/.azure/cliextensions'

Python (Darwin) 3.10.11 (main, Aug 13 2024, 22:14:06) [Clang 15.0.0 (clang-1500.3.9.4)]

### Additional context

When running the `az aks command invoke` command with the `--debug` flag, I can see that the CLI connects to management.azure.com rather than making a direct request to the kubernetes API server; which leads to this command properly working even if I explicitly try to break the DNS by adding the public and private FQDN to my /etc/hosts file pointing to an invalid IP. This is the behavior I am expecting to also happen for the key vault requests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.