Azure / Azure/azure-powershell

New-AzureRmNetworkInterface with -loadbalancerinboundrule fails "A retryable error occurred." if more than 10 rules are applied at once

Open
#4,889 8 comments 0 reactions 0 assignees View on GitHub
act-quality-productivity-squad customer-reported Network - Load Balancer Service Attention
Dominant language
C#
Stars
4.8k
Forks
4.3k
Avg merge
2d 17h
Merged PRs (30d)
51

Description

### Cmdlet(s)
New-AzureRmNetworkInterface

### PowerShell Version
PSVersion 5.1.16299.19

### Module Version
4.4.1 (AzureRM)

### OS Version
BuildVersion
10.0.16299.19

### Description
New-AzureRMNetworkInterface and parameter -loadbalancinginboundrule fails when deploying over 10 NAT rules in same time using PowerShell jobs.

We use Powershell to deploy 1-20 of VM's to Azure. As a part of the deployment we first make Azure public Load Balancer that has 20 predefined rules. Then we have function that contains basic Azure VM setup and it ends to New-AzureRMNetworkInterface cmdlet that joins VM NIC to predefined NAT rule. To get VM's faster, we use jobs to run 1-20 "New-DemoVM" function at the same time.

It seems that if you are trying to do over 10 simultaneous NAT rule configs while using jobs, it usually fails randomly and some of VM's don't get NAT rules configured. 9 rules works everytime, so does 9+9.

### Debug Output
This happens x time

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
429

Headers:
Pragma : no-cache
x-ms-request-id : c7925227-1512-42ff-8986-f2770b0866fe
Strict-Transport-Security : max-age=31536000; includeSubDomains
Cache-Control : no-cache
Server : Microsoft-HTTPAPI/2.0,Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1193
x-ms-correlation-request-id : 44514866-7458-449f-8cf0-2073d9e67f52
x-ms-routing-request-id : WESTEUROPE:20171030T220720Z:44514866-7458-449f-8cf0-2073d9e67f52
Date : Mon, 30 Oct 2017 22:07:20 GMT
Connection : close

Body:
{
"error": {
"code": "RetryableError",
"message": "A retryable error occurred.",
"details": [
{
"code": "ReferencedResourceNotProvisioned",
"message": "Cannot proceed with operation because resource /subscriptions/8d6be875-6644-4eb5-a036-335ef672acf7/resourceGroups/lbtest2/providers/Microsoft.Compute/availabilitySets/Clas
sVMs-AS used by resource VM14-NIC is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is InternalOperation."
}
]
}
}

A retryable error occurred.
StatusCode: 429
ReasonPhrase:
OperationID : '99966a18-ebae-445c-89eb-03553eede365'
+ CategoryInfo : CloseError: (:) [New-AzureRmNetworkInterface], NetworkCloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Network.NewAzureNetworkInterfaceCommand
+ PSComputerName : localhost

DEBUG: AzureQoSEvent: CommandName - New-AzureRmNetworkInterface; IsSuccess - False; Duration - 00:00:01.6202057; Exception - Microsoft.Azure.Commands.Network.Common.NetworkCloudException: A r
etryable error occurred.
StatusCode: 429
ReasonPhrase:
OperationID : '99966a18-ebae-445c-89eb-03553eede365' ---> Microsoft.Rest.Azure.CloudException: A retryable error occurred.
at Microsoft.Azure.Management.Network.NetworkInterfacesOperations.d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.Network.NetworkInterfacesOperations.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.Network.NetworkInterfacesOperationsExtensions.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.Network.NetworkInterfacesOperationsExtensions.CreateOrUpdate(INetworkInterfacesOperations operations, String resourceGroupName, String networkInterfaceName, N
etworkInterface parameters)
at Microsoft.Azure.Commands.Network.NewAzureNetworkInterfaceCommand.CreateNetworkInterface() in D:\workspace\powershell-sign\src\ResourceManager\Network\Commands.Network\NetworkInterface\N
ewAzureNetworkInterfaceCommand.cs:line 428
at Microsoft.Azure.Commands.Network.NewAzureNetworkInterfaceCommand.<>c__DisplayClass104_0.b__0() in D:\workspace\powershell-sign\src\ResourceManager\Network\Commands.Network\Netw
orkInterface\NewAzureNetworkInterfaceCommand.cs:line 251
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(Boolean force, String continueMessage, String processMessage, String target, Action action, Func`1 useShould
Continue) in D:\workspace\powershell-sign\src\Common\Commands.Common\AzurePSCmdlet.cs:line 602
at Microsoft.Azure.Commands.Network.NewAzureNetworkInterfaceCommand.Execute() in D:\workspace\powershell-sign\src\ResourceManager\Network\Commands.Network\NetworkInterface\NewAzureNetworkI
nterfaceCommand.cs:line 260
at Microsoft.Azure.Commands.Network.NetworkBaseCmdlet.ExecuteCmdlet() in D:\workspace\powershell-sign\src\ResourceManager\Network\Commands.Network\Common\NetworkBaseCmdlet.cs:line 51
--- End of inner exception stack trace ---
at Microsoft.Azure.Commands.Network.NetworkBaseCmdlet.ExecuteCmdlet() in D:\workspace\powershell-sign\src\ResourceManager\Network\Commands.Network\Common\NetworkBaseCmdlet.cs:line 55
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord() in D:\workspace\powershell-sign\src\Common\Commands.Common\AzurePSCmdlet.cs:line 639;
DEBUG: Finish sending metric.
DEBUG: 0.07.14 - NewAzureNetworkInterfaceCommand end processing.
DEBUG: 0.07.14 - NewAzureNetworkInterfaceCommand end processing.

**PS C:\tmp> Resolve-AzureRmError**

Message : A retryable error occurred.
StatusCode: 429
ReasonPhrase:
OperationID : '3d52b211-4b1d-4eee-a6f9-b32de9c81636'
StackTrace :
Exception : System.Management.Automation.RemoteException
InvocationInfo : {}
Line :
Position :
HistoryId :

### Script/Steps for Reproduction
If I run this over 10 times everything works:

$frontend = Get-AzureRmLoadBalancerFrontendIpConfig -LoadBalancer $lb | ? Id -Contains $vmname
$backendsubnet = Get-AzureRmVirtualNetworkSubnetConfig -VirtualNetwork $vnet -Name default
$natrule = Get-AzureRmLoadBalancerInboundNatRuleConfig -LoadBalancer $lb -Name $vmname
New-AzureRmNetworkInterface `
-ResourceGroupName $rgname `
-Name $vmname-NIC `
-Location $location `
-Subnet $backendsubnet `
-LoadBalancerInboundNatRule $natrule `
-Force

If I run more than 10 jobs at same time I most likely get 1-5 errors mentioned above
function new-demovm $a, $b, $c, $d {
start-job -argumentlist $a, $b, $c, $d {
param ($a, $b, $c, $d)
$frontend = Get-AzureRmLoadBalancerFrontendIpConfig -LoadBalancer $c | ? Id -Contains $a
$backendsubnet = Get-AzureRmVirtualNetworkSubnetConfig -VirtualNetwork $d -Name default
$natrule = Get-AzureRmLoadBalancerInboundNatRuleConfig -LoadBalancer $c -Name $a
New-AzureRmNetworkInterface `
-ResourceGroupName $b `
-Name $a-NIC `
-Location $a.location `
-Subnet $backendsubnet `
-LoadBalancerInboundNatRule $natrule `
-Force}
}
}
0..$vmcount | % {New-DemoVM -vmname a$_ -rg $b -lb $c -vnet $d}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.