bug: azure-cli 2.60+ - ssh fails using keys generated with az ssh config when run multiple times with service principal authentication
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
### Describe the bug
I have reproduced this issue on both Ubuntu20.04 and RHEL8 images from the Azure Marketplace; as well as Microsoft hosted Azure DevOps Ubuntu Runners.
These versions work: 2.55.0, 2.56.0, 2.57.0, 2.58.0, 2.59.0
Broken: 2.60.0, 2.61.0, 2.62.0
**Steps:**
1. Deploy two target Linux VMs in Azure:
- Image: RHELRaw8LVMGen2
- Identity: SystemAssignedIdentity
- No public IP address (private only)
- Install the AADSSHloginForLinux VM extension
2. Create a new EntraID service principal. Grant the service principal the "virtual machine administrator login" role assignment on each virtual machine.
3. Deploy an ubuntu control host to test using Azure CLI to connect:
- Image: Ubuntu2004 (note, this can be reproduced also on the RHELRaw8LVMGen2 market place image)
- No public IP address
4. Install azure cli, and signin on the control host:
```bash
apt install azure-cli
az login --service-principal --allow-no-subscriptions --tenant --username
```
5. Run this shell script:
```bash
az extension add --name ssh
az ssh config --file ~/.ssh/config --ip
az ssh config --file ~/.ssh/config --ip
ssh -v -o "StrictHostKeyChecking no" "echo hello first host!"
ssh -v -o "StrictHostKeyChecking no" "echo hello second host!"
```
Observe whichever host is configured first using az ssh config works; whichever is configured second will fail with this error:
```error
debug1: sign_and_send_pubkey: no separate private key for certificate "/home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub"
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub' are too open.
```
**The the second host will always fail.**
If you switch the order of the az ssh config commands, the opposite host will fail.
If you revert to Azure-cli 2.59.0 or older, this issue is not present:
```bash
AZCLI_VER=2.59.0
sudo apt remove azure-cli
sudo apt-get install azure-cli=${AZCLI_VER}-1~$(lsb_release -cs)
```
### Related command
az ssh config
### Errors
The command completes successfully, however the key does not work for which whichever host has keys created second.
```bash
debug1: sign_and_send_pubkey: no separate private key for certificate "/home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub"
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub' are too open.
```
### Issue script & Debug output
```bash
cli.knack.cli: Command arguments: ['ssh', 'config', '--file', '/home//.ssh/config', '--ip', '', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [, , ]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'ssh': ['azext_ssh']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: ssh 0.158 1 4 /home//.azure/cliextensions/ssh
cli.azure.cli.core: Total (1) 0.158 1 4
cli.azure.cli.core: Loaded 1 groups, 4 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : ssh config
cli.azure.cli.core: Command table: ssh config
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate []
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home//.azure/commands/2024-07-12.22-19-54.ssh_config.64974.log'.
az_command_data_logger: command args: ssh config --file {} --ip {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x7fb4d54edaf0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x7fb4d5511af0>, .add_cache_arguments at 0x7fb4d54b3940>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [, , .parse_ids_arguments at 0x7fb4d54b38b0>]
az_command_data_logger: extension name: ssh
az_command_data_logger: extension version: 2.0.4
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ComputeManagementClient
cli.azure.cli.core.auth.persistence: build_persistence: location='/home//.azure/service_principal_entries.json', encrypt=False
cli.azure.cli.core.auth.persistence: build_persistence: location='/home//.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /home//.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/
msal.authority: openid_config("https://login.microsoftonline.com//v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com//discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com//v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com//kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azext_ssh.ssh_utils: Running ssh-keygen command ssh-keygen -f /home//.ssh/az_ssh_config//id_rsa -t rsa -q -N
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/
msal.authority: openid_config("https://login.microsoftonline.com//v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com//discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com//v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com//kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes= # REMOVED
cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://pas.windows.net/CheckMyAccess/Linux/.default',), # REMOVED
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 5ac412e3-9ba7-4251-9adc-b72c376956fd
cli.azext_ssh.custom: Generating certificate /home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub
cli.azext_ssh.ssh_utils: Running ssh-keygen command ssh-keygen -L -f /home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub
cli.azext_ssh.ssh_utils: Running ssh-keygen command ssh-keygen -L -f /home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub
Generated SSH certificate /home//.ssh/az_ssh_config//id_rsa.pub-aadcert.pub is valid until 2024-07-12 11:01:38 PM in local time.
cli.azext_ssh.ssh_utils: /home//.ssh/az_ssh_config/ contains sensitive information (id_rsa, id_rsa.pub, id_rsa.pub-aadcert.pub). Please delete it once you no longer need this config file.
cli.knack.cli: Event: CommandInvoker.OnTransformResult [, ]
cli.knack.cli: Event: CommandInvoker.OnFilterResult []
cli.knack.cli: Event: Cli.SuccessfulExecute []
cli.knack.cli: Event: Cli.PostExecute []
az_command_data_logger: exit code: 0
cli.__main__: Command ran in 0.792 seconds (init: 0.151, invoke: 0.641)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3974 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/bin/python3.9 /usr/lib64/az/lib/python3.9/site-packages/azure/cli/telemetry/__init__.py /home//.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.
```
### Expected behavior
I can set up ssh keys for multiple hosts using az ssh config on versions newer than azure-cli 2.60; as works in 2.59.0 and older.
### Environment Summary
```bash
azure-cli 2.62.0
core 2.62.0
telemetry 1.1.0
Extensions:
ssh 2.0.4
Dependencies:
msal 1.28.1
azure-mgmt-resource 23.1.1
```
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.