Azure / Azure/azure-powershell
New-AzSqlSyncGroup returning Long running operation failed with status 'NotFound'.
- Dominant language
- C#
- Stars
- 4.8k
- Forks
- 4.3k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 51
Description
### Description
We are calling this task from Powershell, it has been working for a long time (several years) but since yesterday has started returning the NotFound error.
It looks like the sync group is actually created OK but the response is returning an error.
I've checked the versions of powershell (5.217.1) and AZ (9.3.0) and neither have changed. It looks like an error on the Azure side to me.
### Issue script & Debug output
```PowerShell
DEBUG: 09:53:34 - NewAzureSqlSyncGroup begin processing with ParameterSet '__AllParameterSets'.
DEBUG: 09:53:34 - using account id ''...
DEBUG: [Common.Authentication]: Authenticating using Account: '', environment: 'AzureCloud', tenant: ''
DEBUG: 09:53:34 - [SilentAuthenticator] Calling SharedTokenCacheCredential.GetTokenAsync - TenantId:'', Scopes:'https://management.core.windows.net//.default'
, AuthorityHost:'https://login.microsoftonline.com/', UserId:''
DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34] Found 2 cache accounts and 0 broker accounts
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34] Returning 2 accounts
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] MSAL MSAL.Desktop with assembly version '4.30.1.0'. CorrelationId(4a61b42e-cf7f-4b2a-9db1-11fbd9
134bf8)
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] === AcquireTokenSilent Parameters ===
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] LoginHint provided: False
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Account provided: True
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] ForceRefresh: False
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.core.windows.net//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenSilent
IsConfidentialClient - False
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - 4a61b42e-cf7f-4b2a-9db1-11fbd9134bf8
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] === Token Acquisition (SilentRequest) started:
Authority Host: login.microsoftonline.com
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Access token has expired or about to expire. [Current time (03/08/2023 09:53:34) - Expiration Ti
me (03/07/2023 10:40:24 +00:00) - Extended Expiration Time (03/07/2023 10:40:24 +00:00)]
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Refresh token found in the cache? - True
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:34 - ] Resolving authority endpoints... Already resolved? - TRUE
DEBUG: Request [0f0ead17-b9fb-4cfe-bad8-d44540b5ef7b] POST https://login.microsoftonline.com/7e86ca72-6bd1-4f50-a078-23b4b1f3b691/oauth2/v2.0/token
x-client-SKU:REDACTED
x-client-Ver:REDACTED
x-client-CPU:REDACTED
x-client-OS:REDACTED
x-client-current-telemetry:REDACTED
x-client-last-telemetry:REDACTED
x-ms-lib-capability:REDACTED
client-request-id:REDACTED
return-client-request-id:REDACTED
x-app-name:REDACTED
x-app-ver:REDACTED
x-ms-client-request-id:0f0ead17-b9fb-4cfe-bad8-d44540b5ef7b
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.5.0,(.NET Framework 4.8.4614.0; Microsoft Windows 10.0.17763 )
Content-Type:application/x-www-form-urlencoded
client assembly: Azure.Identity
DEBUG: Response [0f0ead17-b9fb-4cfe-bad8-d44540b5ef7b] 200 OK (01.6s)
Pragma:no-cache
Strict-Transport-Security:REDACTED
X-Content-Type-Options:REDACTED
client-request-id:REDACTED
x-ms-request-id:bc9a1576-7798-441f-9ddf-297c44b46e00
x-ms-ests-server:REDACTED
x-ms-clitelem:REDACTED
X-XSS-Protection:REDACTED
Cache-Control:no-store, no-cache
P3P:REDACTED
Set-Cookie:REDACTED
Date:Wed, 08 Mar 2023 09:53:36 GMT
Content-Length:5280
Content-Type:application/json; charset=utf-8
Expires:-1
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Checking client info returned from the server..
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Saving Token Response to cache..
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Subject not present in Id token
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Saving AT in cache and removing overlapping ATs...
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Looking for scopes for the authority in the cache which intersect with https://management.core.w
indows.net//.default
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Intersecting scope entries count - 1
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Matching entries after filtering by user - 1
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Saving Id Token and Account in cache ...
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Saving RT in cache...
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36] Not writing FRT in ADAL legacy cache.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Fetched access token from host login.microsoftonline.com.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] === Token Acquisition finished successfully. An access token was returned with Expiration Time:
03/08/2023 11:15:38 +00:00 and Scopes https://management.core.windows.net//user_impersonation https://management.core.windows.net//.default
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: ExpiresOn: 2023-03-08T11:15:38.0000000+00:00
DEBUG: ============================ HTTP REQUEST ============================
HTTP Method:
GET
Absolute Uri:
https://management.azure.com/subscriptions/955483f5-efdd-4d87-9cf5-c294a4bfca6e/resourceGroups/rg-rating-test/providers/Microsoft.Sql/servers/dbs-rating-test/databases/db-rating-test/syncGroups/
SyncGroup1?api-version=2020-11-01-preview
Headers:
x-ms-client-request-id : 650e2c09-eff8-495d-ad55-6cee956f8a35
accept-language : en-US
Body:
DEBUG: ============================ HTTP RESPONSE ============================
Status Code:
NotFound
Headers:
Pragma : no-cache
x-ms-request-id : 81e8aedd-93d7-499d-a5b0-8bb01a11cb23
x-ms-ratelimit-remaining-subscription-reads: 11999
x-ms-correlation-request-id : fd115904-bc80-4320-963f-006d3974f7d1
x-ms-routing-request-id : UKSOUTH:20230308T095337Z:fd115904-bc80-4320-963f-006d3974f7d1
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
Cache-Control : no-cache
Date : Wed, 08 Mar 2023 09:53:36 GMT
Server : Microsoft-HTTPAPI/2.0
Body:
{
"error": {
"code": "ResourceNotFound",
"message": "The requested resource of type 'Microsoft.Sql/servers/databases/syncGroups' with name 'SyncGroup1' was not found."
}
}
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36] Found 2 cache accounts and 0 broker accounts
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36] Returning 2 accounts
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] MSAL MSAL.Desktop with assembly version '4.30.1.0'. CorrelationId(0721aeff-f471-409d-a37a-d1a264
4c2135)
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] === AcquireTokenSilent Parameters ===
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] LoginHint provided: False
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Account provided: True
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] ForceRefresh: False
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.core.windows.net//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenSilent
IsConfidentialClient - False
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - 0721aeff-f471-409d-a37a-d1a2644c2135
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] === Token Acquisition (SilentRequest) started:
Authority Host: login.microsoftonline.com
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Access token is not expired. Returning the found cache entry. [Current time (03/08/2023 09:53:36
) - Expiration Time (03/08/2023 11:15:38 +00:00) - Extended Expiration Time (03/08/2023 11:15:38 +00:00)]
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Returning access token found in cache. RefreshOn exists ? False
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] Fetched access token from host login.microsoftonline.com.
DEBUG: (False) MSAL 4.30.1.0 MSAL.Desktop Windows Server 2019 Datacenter [03/08/2023 09:53:36 - ] === Token Acquisition finished successfully. An access token was returned with Expiration Time:
03/08/2023 11:15:38 +00:00 and Scopes https://management.core.windows.net//user_impersonation https://management.core.windows.net//.default
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: ExpiresOn: 2023-03-08T11:15:38.0000000+00:00
DEBUG: ============================ HTTP REQUEST ============================
HTTP Method:
PUT
Absolute Uri:
https://management.azure.com/subscriptions/955483f5-efdd-4d87-9cf5-c294a4bfca6e/resourceGroups/rg-rating-test/providers/Microsoft.Sql/servers/dbs-rating-test/databases/db-rating-test/syncGroups/
SyncGroup1?api-version=2020-11-01-preview
Headers:
x-ms-client-request-id : 650e2c09-eff8-495d-ad55-6cee956f8a35
accept-language : en-US
Body:
{
"properties": {
"conflictResolutionPolicy": "HubWin",
"syncDatabaseId": "",
"hubDatabaseUserName": "DBAdmin",
"hubDatabasePassword": "",
"usePrivateLinkConnection": false
}
}
DEBUG: ============================ HTTP RESPONSE ============================
Status Code:
Accepted
Headers:
Pragma : no-cache
Retry-After : 15
Azure-AsyncOperation : https://management.azure.com/subscriptions/955483f5-efdd-4d87-9cf5-c294a4bfca6e/resourceGroups/rg-rating-test/providers/Microsoft.Sql/locations/uksouth/syncGroupA
zureAsyncOperation/3c903d05-2484-4b79-afd1-21251bf06209?api-version=2020-11-01-preview
x-ms-request-id : 3c903d05-2484-4b79-afd1-21251bf06209
x-ms-ratelimit-remaining-subscription-writes: 1199
x-ms-correlation-request-id : 5af88107-85df-4e05-a1a8-03b23f18c372
x-ms-routing-request-id : UKSOUTH:20230308T095337Z:5af88107-85df-4e05-a1a8-03b23f18c372
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
Cache-Control : no-cache
Date : Wed, 08 Mar 2023 09:53:36 GMT
Location : https://management.azure.com/subscriptions/955483f5-efdd-4d87-9cf5-c294a4bfca6e/resourceGroups/rg-rating-test/providers/Microsoft.Sql/locations/uksouth/syncGroupO
perationResults/3c903d05-2484-4b79-afd1-21251bf06209?api-version=2020-11-01-preview
Server : Microsoft-HTTPAPI/2.0
Body:
{
"operation": "UpsertSyncGroup",
"startTime": "2023-03-08T09:53:37.303Z"
}
DEBUG: ============================ HTTP REQUEST ============================
HTTP Method:
GET
Absolute Uri:
https://management.azure.com/subscriptions/955483f5-efdd-4d87-9cf5-c294a4bfca6e/resourceGroups/rg-rating-test/providers/Microsoft.Sql/locations/uksouth/syncGroupAzureAsyncOperation/3c903d05-2484
-4b79-afd1-21251bf06209?api-version=2020-11-01-preview
Headers:
x-ms-client-request-id : 650e2c09-eff8-495d-ad55-6cee956f8a35
Body:
DEBUG: ============================ HTTP RESPONSE ============================
Status Code:
NotFound
Headers:
Pragma : no-cache
x-ms-failure-cause : gateway
x-ms-request-id : 93d40df4-fc72-4ec5-8531-af6aa667a71b
x-ms-correlation-request-id : 93d40df4-fc72-4ec5-8531-af6aa667a71b
x-ms-routing-request-id : UKSOUTH:20230308T095352Z:93d40df4-fc72-4ec5-8531-af6aa667a71b
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
Cache-Control : no-cache
Date : Wed, 08 Mar 2023 09:53:51 GMT
Body:
{
"error": {
"code": "InvalidResourceType",
"message": "The resource type 'locations' could not be found in the namespace 'Microsoft.Sql' for api version '2020-11-01-preview'. The supported api-versions are '2014-01-01,2014-04-01,2014
-04-01-preview'."
}
}
New-AzSqlSyncGroup : Long running operation failed with status 'NotFound'.
At C:\dev\Rating\devops\CreateSQLDataSyncGroups.ps1:21 char:1
+ New-AzSqlSyncGroup -ResourceGroupName $ResourceGroup -ServerName $Ser ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [New-AzSqlSyncGroup], CloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Sql.DataSync.Cmdlet.NewAzureSqlSyncGroup
DEBUG: AzureQoSEvent: Module: Az.Sql:3.5.1; CommandName: New-AzSqlSyncGroup; PSVersion: 5.1.17763.3770; IsSuccess: False; Duration: 00:00:17.0989993; Exception: Long running operation failed wit
h status 'NotFound'.;
DEBUG: Finish sending metric.
DEBUG: 09:53:52 - NewAzureSqlSyncGroup end processing.
```
### Environment data
```PowerShell
Name Value
---- -----
PSVersion 5.1.17763.3770
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.17763.3770
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
```
### Module versions
```PowerShell
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 2.6.2 Az.Accounts {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount...}
Script 3.5.1 Az.Sql {Add-AzSqlDatabaseToFailoverGroup, Add-AzSqlElasticJobStep, Add-AzSqlElasticJobTarget, Add-AzSqlInstanceKeyVaultKey...}
```
### Error output
```PowerShell
RequestId :
Message : Long running operation failed with status 'NotFound'.
ServerMessage : InvalidResourceType: The resource type 'locations' could not be found in the namespace 'Microsoft.Sql' for api version '2020-11-01-preview'. The supported api-versions are
'2014-01-01,2014-04-01,2014-04-01-preview'. (System.Collections.Generic.List`1[Microsoft.Rest.Azure.CloudError])
ServerResponse : {NotFound}
```
Contributor guide
Assessment
This issue has not been assessed yet.