Azure / Azure/azure-cli

"az rest" throws exception 'Microsoft.Azure.Event.Common.EventServiceException' intermittently

Open
#17,908 1 comment 0 reactions 1 assignee Claimed by @jiasli View on GitHub
feature-request rest
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

**Describe the bug**
I am running a PowerShell script from Azure DevOps pipeline (AzureCLI@2 task) scheduled to run every hour on MS hosted agent 'ubuntu-latest'. This script gets Azure Activity Logs from each subscription (20 subscriptions in total). I am using `az rest` to make call to Azure management api for activity logs. It works all fine but intermittently fails with below exception. Every time it fails for different subscription so not related to any specific subscription. Attached is debug logs
[ErrorDetails.txt](https://github.com/Azure/azure-cli/files/6391513/ErrorDetails.txt)
.

`ERROR: cli.azure.cli.core.azclierror: Internal Server Error({"code":"InternalServerError","message":"Exception of type 'Microsoft.Azure.Event.Common.EventServiceException' was thrown."})`

**To Reproduce**
Make `Get` requests to azure management api using `az rest` in loop for each subscription. I wrote below function for each subscription to fetch activity event logs.

```
function Get-ActivityLogs(
[Parameter(Mandatory = $true)]
[string]
$SubscriptionID,

[Parameter(Mandatory = $true)]
[string]
$EventsStartTimestamp,

[Parameter(Mandatory = $true)]
[string]
$EventsEndTimestamp
){
$apiUrl = "https://management.azure.com/subscriptions/$SubscriptionID/providers/microsoft.insights/eventtypes/management/values?`$" `
+ "filter=eventTimestamp ge '$EventsStartTimestamp' and eventTimestamp le '$EventsEndTimestamp' " `
+ "and eventChannels eq 'Admin, Operation' and resourceTypes eq 'Microsoft.Compute/VirtualMachines' " `
+ "and levels eq 'Informational'&api-version=2017-03-01-preview"

$results = $()

while ($apiUrl)
{

$response = az rest `
--method get `
--uri `"$apiUrl`" `
| ConvertFrom-Json

if ($LASTEXITCODE -ne 0)
{

Write-Error "Failed to retrieve activity log events for the '$SubscriptionId' subscription."
}

$results += $response.value

$apiUrl = $response.nextLink
}

return $results

}

```

**Expected behavior**

**Environment summary**

Task : Azure CLI
Description : Run Azure CLI commands against an Azure subscription in a PowerShell Core/Shell script when running on Linux agent or PowerShell/PowerShell Core/Batch script when running on Windows agent.
Version : 2.1.0
Author : Microsoft Corporation

**Additional context**

We run `az upgrade -y` every time to make sure latest version be available on agent. Below are some debug logs :
```

2021-04-27T19:11:30.9389537Z The following packages will be upgraded:
2021-04-27T19:11:30.9398356Z azure-cli
2021-04-27T19:11:31.0650148Z 1 upgraded, 0 newly installed, 0 to remove and 25 not upgraded.
2021-04-27T19:11:31.0651513Z Need to get 60.3 MB of archives.
2021-04-27T19:11:31.0652673Z After this operation, 1024 B of additional disk space will be used.
2021-04-27T19:11:31.0654276Z Get:1 https://packages.microsoft.com/repos/azure-cli focal/main amd64 azure-cli all 2.22.1-1~focal [60.3 MB]
2021-04-27T19:11:32.8862826Z Fetched 60.3 MB in 1s (46.1 MB/s)
2021-04-27T19:11:33.2919982Z (Reading database ...
2021-04-27T19:11:33.2920534Z (Reading database ... 5%
2021-04-27T19:11:33.2920969Z (Reading database ... 10%
2021-04-27T19:11:33.2921383Z (Reading database ... 15%
2021-04-27T19:11:33.2921771Z (Reading database ... 20%
2021-04-27T19:11:33.2922155Z (Reading database ... 25%
2021-04-27T19:11:33.2922542Z (Reading database ... 30%
2021-04-27T19:11:33.2922943Z (Reading database ... 35%
2021-04-27T19:11:33.2923326Z (Reading database ... 40%
2021-04-27T19:11:33.2923713Z (Reading database ... 45%
2021-04-27T19:11:33.2924100Z (Reading database ... 50%
2021-04-27T19:11:33.6273796Z (Reading database ... 55%
2021-04-27T19:11:34.2340801Z (Reading database ... 60%
2021-04-27T19:11:35.1346949Z (Reading database ... 65%
2021-04-27T19:11:35.8807392Z (Reading database ... 70%
2021-04-27T19:11:36.7714171Z (Reading database ... 75%
2021-04-27T19:11:37.9047446Z (Reading database ... 80%
2021-04-27T19:11:39.0977840Z (Reading database ... 85%
2021-04-27T19:11:40.0876900Z (Reading database ... 90%
2021-04-27T19:11:41.0008461Z (Reading database ... 95%
2021-04-27T19:11:41.0009925Z (Reading database ... 100%
2021-04-27T19:11:41.0011249Z (Reading database ... 227031 files and directories currently installed.)
2021-04-27T19:11:41.0120133Z Preparing to unpack .../azure-cli_2.22.1-1~focal_all.deb ...
2021-04-27T19:11:41.0171693Z Unpacking azure-cli (2.22.1-1~focal) over (2.22.0-1~focal) ...
2021-04-27T19:12:00.7151872Z Setting up azure-cli (2.22.1-1~focal) ...
2021-04-27T19:12:04.2621456Z WARNING: Upgrading extensions
2021-04-27T19:12:04.2624326Z WARNING: Checking update for azure-devops
2021-04-27T19:12:08.3058376Z WARNING: No updates available for 'azure-devops'. Use --debug for more information.
2021-04-27T19:12:08.3775512Z WARNING: Upgrade finished.You can enable auto-upgrade with 'az config set auto-upgrade.enable=yes'. More details in https://docs.microsoft.com/cli/azure/update-azure-cli#automatic-update
2021-04-27T19:12:12.8093905Z WARNING: The installed extension 'resource-graph' is in preview.

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.