Azure / Azure/azure-powershell
Invoke-AzDataFactoryV2Pipeline does not start new instance of the pipeline if it has just been triggered
- Dominant language
- C#
- Stars
- 4.8k
- Forks
- 4.3k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 51
Description
### Description
`Invoke-AzDataFactoryV2Pipeline` is being called from multiple ThreadJobs, simultaneously, for the same pipeline, with different execution parameters.
What happens sometimes is that this line
`$pipelineRunId = Invoke-AzDataFactoryV2Pipeline -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineName $PipelineName -Parameter $ExecParams`
will not invoke a new pipeline but instead returns the pipelineRunId from the just previously triggered execution, even if the parameters don't match, if the timing is just right (or wrong, in this case).
It does not throw any error at these step, only our internal validation did when it realized it was not the expected pipeline run.
I created a test ADF pipeline that just waits for the given input of seconds

and then I used this script to reproduce the issue
```
param(
$ResourceGroupName,
$DataFactoryName,
$PipelineName,
$ConcurrentCalls
)
Get-Job | Remove-Job -Force
$scriptBlock = {
try{
$ResourceGroupName = $using:ResourceGroupName
$DataFactoryName = $using:DataFactoryName
$PipelineName = $using:PipelineName
$i = $using:i
$randomWait = (Get-Random -Maximum 5 -Minimum 3)
$ExecParams = @{ }
$ExecParams.Add("WaitTime", "$randomWait")
$ExecParams.Add("JobId", "$i")
Write-Host "START: $(get-date)"
Write-Host "ExecParams:"
$ExecParams | Out-String | Write-Host
# Invoking ADF Pipeline
$pipelineRunId = Invoke-AzDataFactoryV2Pipeline -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineName $PipelineName -Parameter $ExecParams
Write-Host "PROGRESS: pipelineRunId: $pipelineRunId"
Write-Host "DONE: $(get-date)"
}
catch {
throw
}
}
for ($i = 1; $i -le $ConcurrentCalls; $i++) {
$job = Start-ThreadJob -Name "Job for iteration $i" -ScriptBlock $scriptBlock
}
Start-Sleep 1
while ($true) {
$completedJobs = Get-Job -State Completed
$runningJobs = Get-Job -State Running
$notStartedJobs = Get-Job -State NotStarted
foreach ($job in $completedJobs) {
Write-Host "Found completed jobs:" -ForegroundColor Cyan
Receive-Job -Id $job.Id
Write-Host "Cleanup completed jobs" -ForegroundColor Cyan
Remove-Job -Id $job.Id
}
# once no more running jobs, exit
if(($runningJobs.count + $notStartedJobs.Count) -eq 0){
break
}
Start-Sleep 1
}
```
It's not consistent but, as shown below, it can return the same pipelineRunId for different jobs (and each job has a different parameter and should trigger a different pipeline execution)
As a work around, I created a function `Invoke-AzDataFactoryV2PipelineSafely` that will only "accept" the pipelineRunId if one distinctive parameter is the same from the call and from the info of the returned pipeline.

### Issue script & Debug output
```PowerShell
& "$rootPath\InvokeAzDataFactoryV2Pipeline_repro.ps1" -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineName $PipelineName -ConcurrentCalls 3
Found completed jobs:
START: 07/23/2024 15:07:55
ExecParams:
Name Value
---- -----
WaitTime 4
JobId 1
PROGRESS: pipelineRunId: 4033f90c-2a3a-4768-a0c4-76ff399dfb0e
DONE: 07/23/2024 15:07:56
Cleanup completed jobs
Found completed jobs:
START: 07/23/2024 15:07:55
ExecParams:
Name Value
---- -----
WaitTime 4
JobId 2
PROGRESS: pipelineRunId: ac5b4fc5-d978-476e-a491-76b74846d7ab
DONE: 07/23/2024 15:07:56
Cleanup completed jobs
Found completed jobs:
START: 07/23/2024 15:07:55
ExecParams:
Name Value
---- -----
WaitTime 4
JobId 3
PROGRESS: pipelineRunId: ac5b4fc5-d978-476e-a491-76b74846d7ab
DONE: 07/23/2024 15:07:56
Cleanup completed jobs
```
### Environment data
```PowerShell
Name Value
---- -----
PSVersion 7.4.3
PSEdition Core
GitCommitId 7.4.3
OS Microsoft Windows 10.0.19045
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
```
### Module versions
```PowerShell
ModuleType Version PreRelease Name PSEdition ExportedCommands
---------- ------- ---------- ---- --------- ----------------
Script 3.0.2 Az.Accounts Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave…}
Script 2.13.1 Az.Accounts Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave…}
Script 1.1.4 Az.AnalysisServices Core,Desk {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysisServicesServer, Remove-AzAnalysisServicesServer…}
Script 1.18.5 Az.DataFactory Core,Desk {Add-AzDataFactoryV2DataFlowDebugSessionPackage, Add-AzDataFactoryV2TriggerSubscription, Get-AzDataFactory, Get-AzDataFactoryActivityWindow…}
Script 4.9.3 Az.KeyVault Core,Desk {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertificateOperation, Get-AzKeyVaultCertificateOperation…}
Script 6.5.3 Az.Resources Core,Desk {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssignment…}
Script 5.10.1 Az.Storage Core,Desk {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAccountKey…}
Script 1.3.0 azure.datafactory.tools Desk {Publish-AdfV2FromJson, Publish-AdfV2UsingArm, Get-AdfFromService, New-AdfPublishOption…}
```
### Error output
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.