Encountered Transient Failure with Status Code : POST during method 423 when applying site template (search configuration)

Hi all, when using pnp provisioning from Azure Function we are facing an intermittent issue in the "web.ApplyProvisioningTemplate" method. Full error message / callstack:

Exception has occurred: CLR/Microsoft.SharePoint.Client.ServerException
Exception thrown: 'Microsoft.SharePoint.Client.ServerException' in PnP.Framework.dll: 'Encountered Transient Failure with Status Code : POST during method 423.'
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
   at Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryToServerAsync>d__53.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.SharePoint.Client.ClientRequest.<ExecuteQueryAsync>d__39.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.SharePoint.Client.ClientRuntimeContext.<ExecuteQueryAsync>d__57.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.SharePoint.Client.ClientContext.<ExecuteQueryAsync>d__23.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.SharePoint.Client.ClientContextExtensions.<ExecuteQueryImplementation>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.SharePoint.Client.ClientContextExtensions.ExecuteQueryRetry(ClientRuntimeContext clientContext, Int32 retryCount, String userAgent)
   at Microsoft.SharePoint.Client.SearchExtensions.SetSearchConfigurationImplementation(ClientRuntimeContext context, SearchObjectLevel searchObjectLevel, String searchConfiguration)
   at Microsoft.SharePoint.Client.SearchExtensions.SetSearchConfiguration(Site site, String searchConfiguration)
   at PnP.Framework.Provisioning.ObjectHandlers.ObjectSearchSettings.ProvisionObjects(Web web, ProvisioningTemplate template, TokenParser parser, ProvisioningTemplateApplyingInformation applyingInformation)
   at PnP.Framework.Provisioning.ObjectHandlers.SiteToTemplateConversion.ApplyRemoteTemplate(Web web, ProvisioningTemplate template, ProvisioningTemplateApplyingInformation provisioningInfo, Boolean calledFromHierarchy, TokenParser tokenParser)
   at Microsoft.SharePoint.Client.WebExtensions.ApplyProvisioningTemplate(Web web, ProvisioningTemplate template, ProvisioningTemplateApplyingInformation applyingInformation)

From the callstack seems to be an issue with error handling of the "SetSearchConfigurationImplementation" method but it works most of the time.

Our search configuration is the following:

<pnp:ProvisioningTemplate ID="site001" Version="1" BaseSiteTemplate="SITEPAGEPUBLISHING#0" Scope="RootSite">

    <pnp:WebSettings RequestAccessEmail="" NoCrawl="false" WelcomePage="SitePages/Home.aspx" SiteLogo="{Site}/SiteAssets/logo.png" AlternateCSS="" MasterPageUrl="{masterpagecatalog}/seattle.master" CustomMasterPageUrl="{masterpagecatalog}/seattle.master" CommentsOnSitePagesDisabled="true" QuickLaunchEnabled="true" MembersCanShare="true" HorizontalQuickLaunch="true" SearchScope="Hub" SearchBoxInNavBar="Inherit" HubSiteUrl="{hosturl}{parameter:RootSiteUrl}"  />
    <pnp:SiteSettings AllowDesigner="true" AllowCreateDeclarativeWorkflow="true" AllowSaveDeclarativeWorkflowAsTemplate="true" AllowSavePublishDeclarativeWorkflow="true" SearchBoxInNavBar="Inherit" SocialBarOnSitePagesDisabled="true" />

</pnp:ProvisioningTemplate>

Our workaround is limit the batch size of the Azure Function to 1 using the following host.json configuration:

{
    "version": "2.0",
    "extensions": {
        "queues": {
            "batchSize": 1
        }
    }
}

Start by reading SetSearchConfigurationImplementation and the ObjectSearchSettings.ProvisionObjects call in the stack trace. Reproduce ApplyProvisioningTemplate with the supplied search configuration, comparing Azure Function queue batch sizes of 1 and the normal setting. Done should explain the intermittent POST failure and establish reliable provisioning without relying on the workaround.