Azure / Azure/azure-powershell

Restore-AzWebAppSnapshot fails when large number of resources exist (sometimes)

Open
#26,268 3 comments 0 reactions 0 assignees View on GitHub
act-observability-squad App Services bug customer-reported Service Attention
Dominant language
C#
Stars
4.8k
Forks
4.3k
Avg merge
2d 17h
Merged PRs (30d)
51

Description

### Description

Restore-AzWebAppSnapshot can fail when a large number of resources exist in the subscription.
Restore-AzWebAppSnapshot calls the "resources" API which returns paginated results.

If the Web App of the Snapshot is not returned on the first "page" of results, the restore operation will fail with an InternalServerError

**Note:** a "filter" is passed to this API call, however, it appears that the Resources API is first retrieving all results, then only filtering using the first page.

I suspect the above as I have tested the API call using the Azure Resource Explorer (https://resources.azure.com/subscriptions/[subscription-guid-redacted]/resources)
We have 24 different apps and only have consistent issues restoring 5 of them with this call.

All 5 of the affected web apps do not appear on the first "page" of results in Azure Resource Explorer, while all working apps DO appear on the first page.

In addition, we have one web app that works _some_ of the time.
Refreshing the results in Azure Resource Explorer shows that sometimes this app appears on the first page and sometimes it does not.

**Note:** In the debug output below you will see the call to the Resources API
https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01

Observe it returns an empty array with a "nextLink"

### Issue script & Debug output

```powershell
$snapshot = (Get-AzWebAppSnapshot -Name $WebAppName -ResourceGroupName $ResourceGroupName)[0]

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots?api-version=2021-01-15

Headers:
x-ms-client-request-id : 8b732e64-9590-447f-8ef0-a81b8b8f317c
Accept-Language : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control : no-cache
Pragma : no-cache
ETag : "1DB0EE97209AB0B"
Strict-Transport-Security : max-age=31536000; includeSubDomains
x-ms-request-id : 86bd8e45-aff8-4a31-8c31-059e0458e97e
X-AspNet-Version : 4.0.30319
X-Powered-By : ASP.NET
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-correlation-request-id : d27badb3-c7e0-474e-9c7e-44c996ba1411
x-ms-routing-request-id : EASTUS:20241010T204514Z:d27badb3-c7e0-474e-9c7e-44c996ba1411
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: E117A506F88B46559CD0664687396A12 Ref B: MNZ221060610029 Ref C: 2024-10-10T20:45:13Z
Date : Thu, 10 Oct 2024 20:45:14 GMT

Body:
{
"value": [
{
"id": "/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots/2024-09-11T00_25_46_4226984",
"name": "2024-09-11T00:25:46.4226984",
"type": "Microsoft.Web/sites/snapshots",
"properties": {
"time": "2024-09-11T00:25:46.4226984"
}
},
...
]
}
```
```powershell
Restore-AzWebAppSnapshot -Name $TargetWebAppName -ResourceGroupName $TargetResourceGroupName -InputObject $snapshot -Force -RecoverConfiguration -Debug

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01

Headers:
x-ms-client-request-id : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control : no-cache
Pragma : no-cache
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-request-id : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-correlation-request-id : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-routing-request-id : EASTUS:20241010T204621Z:52ce3c1d-ac04-4795-bd9a-84a4e949f186
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: E4B43C8E16E345328AA4FB9B3127D5CB Ref B: MNZ221060618037 Ref C: 2024-10-10T20:46:21Z
Date : Thu, 10 Oct 2024 20:46:20 GMT

Body:
{
"value": [],
"nextLink": "https://management.azure.com/subscriptions/[redacted]/resources?%24filter=resourceType+eq+%27Microsoft.Web%2fsites%27+and+name+eq+%27[redacted]%27&api-version=2016-09-01&%24skiptoken=rZFNa4NAEED%2fi2cPm9VcAj20XWMNzoplzXb3JkbqR1TQFaOh%2f73bIC2BhID0OgNvHvPORp2elJ%2fXZWdszkbdtCpL0lq18bHXEyMdd738yBDj2zHG%2b9ErmpxWoSU4DHQCRXk00hEhwUrb584JsFCiEEhU0TogyRowDF6NngzT6Jr%2bMVvwENMpQcCelWRg0VeEAu6MPvscBHcUMAeBC3ZAwKYsQjM7jTvVd%2fiOsHTDUZJwLXmkaLU9Sg2FybkICyYUJTKnTOayEBYtDtUMffQHgWUGPFwBdpR2Wf24SlIO%2bg8ocDUWvxd6d5LEmWSxy65c76gK7NlAIksWiQrYSwm5Vq32lc%2b9AapIBS6sxJRYAckyIIf8wvwy%2fyoG9VtzPOiW8xDitkxbfe067ea%2fwprXWW9yl0Q156Q3gUuCmr85byKXxJwd7%2fCWhPwG"
}

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
POST

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/restoreSnapshot?api-version=2021-01-15

Headers:
x-ms-client-request-id : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language : en-US

Body:
{
"properties": {
"snapshotTime": "2024-10-10T19:49:23",
"recoverySource": {},
"overwrite": true,
"recoverConfiguration": true,
"ignoreConflictingHostNames": true,
"useDRSecondary": false
}
}

...

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
InternalServerError

Headers:
Cache-Control : no-cache
Cache-Control : no-cache
Pragma : no-cache
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-AspNet-Version : 4.0.30319
X-Powered-By : ASP.NET
x-ms-failure-cause : service
x-ms-ratelimit-remaining-subscription-writes: 199
x-ms-ratelimit-remaining-subscription-global-writes: 2999
x-ms-request-id : e65257ab-3bba-4119-8811-d379bf759616
x-ms-correlation-request-id : e65257ab-3bba-4119-8811-d379bf759616
x-ms-routing-request-id : EASTUS:20241010T204644Z:e65257ab-3bba-4119-8811-d379bf759616
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: D5FB6060669A470A849BE4CBC9E966BF Ref B: MNZ221060610047 Ref C: 2024-10-10T20:46:44Z
Date : Thu, 10 Oct 2024 20:46:43 GMT

Body:
{
"Message": "An error has occurred."
}
```

### Environment data
```
Name Value
---- -----
PSVersion 7.2.23
PSEdition Core
GitCommitId 7.2.23
OS Microsoft Windows 10.0.22621
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
```

### Module versions

```
ModuleType Version PreRelease Name ExportedCommands
---------- ------- ---------- ---- ----------------
Script 3.0.2 Az.Accounts {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script 5.1.0 Az.KeyVault {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContact, Add-AzKeyVaultKey, Add-AzKe…
Script 6.13.0 Az.Resources {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script 1.2.2 Az.TrafficManager {Add-AzTrafficManagerCustomHeaderToEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, A…
Script 3.1.2 Az.Websites {Add-AzWebAppAccessRestrictionRule, Add-AzWebAppTrafficRouting, Edit-AzWebAppBackupConfig…
```

### Error output

```
Message : Operation returned an invalid status code 'InternalServerError'
StackTrace : at Microsoft.Azure.Management.WebSites.WebAppsOperations.BeginRestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,
SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.Management.WebSites.WebAppsOperations.RestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,
SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshotAsync(IWebAppsOperations operations, String
resourceGroupName, String name, SnapshotRestoreRequest restoreRequest, CancellationToken cancellationToken)
at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshot(IWebAppsOperations operations, String
resourceGroupName, String name, SnapshotRestoreRequest restoreRequest)
at Microsoft.Azure.Commands.WebApps.Utilities.WebsitesClient.RestoreSnapshot(String resourceGroupName, String webSiteName, String
slotName, SnapshotRestoreRequest restoreReq)
at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.<>c__DisplayClass17_0.b__0()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(Boolean force, String continueMessage, String
processMessage, String target, Action action, Func`1 useShouldContinue)
at Microsoft.Azure.Commands.ResourceManager.Common.AzureRMCmdlet.ConfirmAction(Boolean force, String continueMessage, String
processMessage, String target, Action action)
at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.b__3_0(T c)
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception : Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException
InvocationInfo : {Restore-AzWebAppSnapshot}
Line : Restore-AzWebAppSnapshot -Name [redacted] -ResourceGroupName [redacted] -InputObject $snapshot -Force
-RecoverConfiguration -Debug
```

Contributor guide

Open the contributing guide

Research direction

Start at Restore-AzWebAppSnapshot and the WebsitesClient.RestoreSnapshot stack-trace entry, then inspect the GET to the subscription resources API shown in the debug output. Follow the returned nextLink when the filtered web app is not on the first page; done means the restore no longer fails for paginated resource results and the reported scenario is covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, powershell
Domain
api, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.