What-If deployment poor performance at scale

Overview

We use the what-if deployment for Azure Policies in our tenant. We grown our policy footprint in the last 12 months and have nearly 350 custom definitions (before this it was a bit under 200), 321 of them are at a single scope. When we do the what-if deployment we group the RESTAPI calls by scope. Our Tenant Root Group has all 321. After repeated tickets with MS over the years, we had pushed the manual timeout limit to 1hr which was being hit often so our pipeline would automatically do the deployment which ran in under 2 minutes most times.

When we had under 200 definitions, we had the timeout set to 30mins and rarely hit it. The advice from support tickets was simply to wait for a success or failure reply.

Last week we decided to try batching the requests instead. This worked much better and takes under 1 minute. We have grouped them in batches of 100 and run the in a foreach-object -parallel loop so the tenant root group is still processing all 321 policies at the same time but it's split across 4 different requests. I have been going through the documentation and not found this limitation. Are you able to provide any guidance on why this was happening?

All of the policy definition templates are written in ARM and we use PowerShell to collate the JSON into a single deployment JSON

Documentation

This section details documentation we're following:
ARM template deployment what-if operation

Deployments - What If

Deployments - Create Or Update At Scope

For all of the above we use api version 2025-04-01

Environment Information

DevOps Agent: Windows Self-Hosted
DevOps Agent Version: 4.269.0
DevOps Task: AzurePowershell@5
DevOps Task version: 5.276.1
Note: We do not hard set the task version so it updates whenever there's a new release

PowerShell:
$PSVersionTable.PSVersion

Major Minor Patch PreReleaseLabel BuildLabel

7 5 0

Code Workflow:

• Get Files
• Go through all files and make changes based on target azure tenant
• Create variable for deployment
• Run the what-if and actual deployment in parallel
• If what-if returns no-change then the deployment for that scope ends and sends log to variable
• If what-if determines a change, log the before/after/delta
• Do deployment
• Show log output from job

It is quite a simple deployment and we use restapi for it as we've found it works best. Though it is very confusing why we were being hit with such poor performance

Start with the ARM template deployment what-if documentation and the Deployments What-If and Create Or Update At Scope REST API entry points, using API version 2025-04-01. Compare the single 321-policy request with the four batched requests and document the cause, applicable limits, or recommended guidance for this performance difference.