Get/Remove-AzRoleAssignment is not working reliably

See also comment on issue 10854 (https://github.com/Azure/azure-powershell/issues/10854).

I have got the same problem, the command Get-AzRoleAssignment is very instable and the same problem is occurring regularly with Remove-AzRoleAssignment. The remove command is first calling the same REST API for getting the existing role assignments as the get command. With $DebugPreference = 'Continue' I was not able now to reproduce the problem with the remove command, but will keep trying. Without debug output I get the error regularly, looks like that:

```
Remove-AzRoleAssignment : The provided information does not map to a role assignment.
At C:\Users\username\Dev\Script.ps1:353 char:13
+ Remove-AzRoleAssignment -ObjectId $user.Id -RoleDefinitio ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Remove-AzRoleAssignment], KeyNotFoundException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.RemoveAzureRoleAssignmentCommand

Error$ entry:
Message : The provided information does not map to a role assignment.
StackTrace : at Microsoft.Azure.Commands.Resources.Models.Authorization.AuthorizationClient.RemoveRoleAssignment(FilterRoleAssignmentsOptions options, String subscriptionId)
at Microsoft.Azure.Commands.Resources.RemoveAzureRoleAssignmentCommand.<>c__DisplayClass48_0.b__1()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(String processMessage, String target, Action action)
at Microsoft.Azure.Commands.Resources.RemoveAzureRoleAssignmentCommand.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception : System.Collections.Generic.KeyNotFoundException
InvocationInfo : {Remove-AzRoleAssignment}
Line : Remove-AzRoleAssignment -ObjectId $user.Id -RoleDefinitionName 'Virtual Machine Contributor' -ResourceName $vmInfo.Name -ResourceGroupName $vmInfo.ResourceGroupName -ResourceType $vmInfo.ResourceType

Position : At C:\Users\username\Script.ps1:357 char:13
+ Remove-AzRoleAssignment -ObjectId $user.Id -RoleDefinitio ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId : 212
```

Sometimes I also get the following error message:

```
Remove-AzRoleAssignment : Operation returned an invalid status code 'NoContent'
At C:\Users\username\Dev\Script.ps1:353 char:13
+ Remove-AzRoleAssignment -ObjectId $user.Id -RoleDefinitio ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Remove-AzRoleAssignment], CloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.RemoveAzureRoleAssignmentCommand

```

I have to say that I'm removing the role assignment again after performing some commands on a VM just about 20 seconds later again. But the access for the other commands on the VM are working fine (so the role assignment is working) and in the Azure Portal I'm also seeing it immediately.

I tried both with -Scope and individual arguments (ResourceGroupName, ResourceName, ResourceType), I also selected the subscription upfront. Same sporadic but very frequent behavior.

The attached log

[Get-AzRoleAssignment-DebugOutput.txt](https://github.com/Azure/azure-powershell/files/7195669/Get-AzRoleAssignment-DebugOutput.txt)

shows that it seems to have successfully called REST API (2020-04-01-preview). But the return value of the command was null without any error message.

The Resolve-AzError command has given the following output after the error happened:
[Resolve-AzError-Output.txt](https://github.com/Azure/azure-powershell/files/7196000/Resolve-AzError-Output.txt)

After a while it sometimes works again, doing nothing in between, or checking the role assignment in Azure Portal, or calling the command with another set of parameters in between...

- - - - -
Further, the calls seems to be inefficient:

Also, the commend call a second roleDefinitions REST API that gets a lot of information which is not wanted and causes a lot of probably superfluous data transfer.

The remove command even does 4 API calls:

1. Get role assignment (superfluous? existence can be checked in delete call):
GET https://management.azure.com//subscriptions/7a87.../resourceGroups/VSTSAgents/providers/Microsoft.Compute/virtualMachines/Ubuntu1/providers/Microsoft.Authorization/roleAssignments?$filter=principalId eq '25c3...'&api-version=2020-04-01-preview

2. Get all role definitions (superfluous? specified as argument):
GET https://management.azure.com//subscriptions/7a87.../resourceGroups/VSTSAgents/providers/Microsoft.Compute/virtualMachines/Ubuntu1/providers/Microsoft.Authorization/roleDefinitions?api-version=2018-01-01-preview

3. Get user object (superfluous? specified as argument):
POST https://graph.windows.net/cfd2.../getObjectsByObjectIds?api-version=1.6

4. Delete the role assignment:
DELETE https://management.azure.com//subscriptions/7a87.../resourceGroups/VSTSAgents/providers/Microsoft.Compute/virtualMachines/Ubuntu1/providers/Microsoft.Authorization/roleAssignments/ab64...?api-version=2020-04-01-preview

_Originally posted by @thgossler in https://github.com/Azure/azure-powershell/issues/10854#issuecomment-922884616_

Start with the linked issue 10854, the attached Get-AzRoleAssignment debug output, and the Resolve-AzError output. Reproduce repeated Get-AzRoleAssignment and Remove-AzRoleAssignment calls around the reported 20-second interval, comparing behavior with and without debug output and tracing the listed REST requests. Done means the sporadic failures and unnecessary request behavior have a confirmed cause and reliable handling.

Written by the indexing model from the issue text.