Azure / Azure/AzureLocal-Supportability

Azure CLI fails on Azure Local nodes with "exit status 1" when run using 32-bit cmd.exe

Open
#228 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
78
Forks
60
Avg merge
1d 6h
Merged PRs (30d)
5

Description

Originally reported here: https://github.com/Azure/azure-sdk-for-go/issues/25722

**Bug description**
We have a CLI extension `arcappliance` which calls into a Go binary for some commands. We started using `AzureCLICredential` from `azure-sdk-for-go` in the Go binary to perform ARM operations. `AzureCLICredential` internals calls Azure CLI via `cmd.exe` to get the token. This works in most cases, but we started seeing issues in Azure Local node environments.

This is the error we would see:
```
AzureCLICredential: exit status 1
```

Note: Azure Local uses a 32-bit build of Azure CLI. This is discussed below.

**Repro steps**
1. Create the following Go program in `cli_cred_repro.go`:

cli_cred_repro.go

```go
package main

import (
"context"
"fmt"

"github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
)

func main() {
creds, err := azidentity.NewAzureCLICredential(nil)
if err != nil {
fmt.Printf("Failed to create credential: %v\n", err)
return
}
fmt.Println("Successfully created credential")

// Uncomment to apply the workaround
// err = os.Setenv("PATH", "C:\\Windows\\Sysnative;"+os.Getenv("PATH"))
// if err != nil {
// fmt.Printf("ERROR: failed to set PATH environment variable: %s\n", err.Error())
// return
// }

token, err := creds.GetToken(context.Background(), policy.TokenRequestOptions{
Scopes: []string{cloud.AzurePublic.Services[cloud.ResourceManager].Audience + "/.default"},
})
if err != nil {
fmt.Printf("Failed to get token: %v\n", err)
return
}

fmt.Printf("Successfully retrieved token with expiry: %s\n", token.ExpiresOn.String())
}
```

2. Compile both Windows 64 bit and 32 bit binaries:
```
$ GOOS=windows GOARCH=amd64 go build -o cli_cred_repro.exe cli_cred_repro.go
$ GOOS=windows GOARCH=386 go build -o cli_cred_repro-32bit.exe cli_cred_repro.go
```

3. Copy the binaries to an Azure Local node and execute them. The 32-bit version will fail until you uncomment the workaround code:
```
[v-host1]: PS C:\> .\cli_cred_repro.exe
Successfully created credential
Successfully retrieved token with expiry: 2025-12-05 01:34:41 +0000 UTC

[v-host1]: PS C:\> .\cli_cred_repro-32bit.exe
Successfully created credential
Failed to get token: AzureCLICredential: exit status 1
```

**Expected behavior**
The 32-bit program should get a token from `AzureCLICredential` successfully, which requires running Azure CLI in a 32-bit `cmd.exe` successfully.

**Environment (please complete the following information):**
- Build [release version, e.g. 10.2405.0.24]: 12.2601.1002.32 (not build specific)
- One-node or multi-node: The issue happens on individual nodes
- Production or non-production: All
- Region [e.g. East US]: All

**Cause**
Azure Local currently uses a 32-bit build of Azure CLI, even though it runs under 64-bit Windows. I found that the 32-bit `cmd.exe` (located at `C:\Windows\SysWOW64\cmd.exe`) on Azure Local nodes is unable to run any scripts, including Azure CLI.

With a script at `C:\test.cmd` that only contains `@echo "test"`, I observed that the 64-bit `cmd.exe` could execute it but not the 32-bit `cmd.exe`:
```
[v-host1]: PS C:\> cmd.exe /c "C:\test.cmd"
"test"

[v-host1]: PS C:\> C:\Windows\SysWOW64\cmd.exe /c "C:\test.cmd"

[v-host1]: PS C:\> $LASTEXITCODE
1
```

Since `az` on Windows runs a script called `az.cmd`, this means Azure CLI will fail to run under the 32-bit `cmd.exe`. I don't know why this is the behavior, but it explains why `AzureCLICredential` fails with no output other than `exit status 1`.

On my local machine I am able to run the script successfully with the 32-bit `cmd.exe`, so it is unclear how Azure Local is different.

**Workaround**
I'm not sure what the long-term fix is (likely it is in Azure Local), but there is a work around.

The Go program can prepend `C:\Windows\Sysnative;` to `PATH` before `AzureCLICredential` is used. This will cause `cmd.exe` to resolve to the 64-bit version instead of 32-bit.

```go
if runtime.GOOS == "windows" && runtime.GOARCH == "386" {
err := os.Setenv("PATH", "C:\\Windows\\Sysnative;"+os.Getenv("PATH"))
if err != nil {
return err
}
}
```

**Screenshots**
N/A

**Correlation ID**
[Collect logs](https://learn.microsoft.com/azure-stack/hci/manage/get-support-for-deployment-issues#perform-standalone-log-collection) and share the correlation ID here.
N/A

**Additional context**
See above "Cause" and "Workaround" sections.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.