Azure / Azure/azure-powershell

Invoke-AzDataFactoryV2Pipeline does not start new instance of the pipeline if it has just been triggered

Open
#25,622 0 comments 0 reactions 0 assignees View on GitHub
bug customer-reported needs-triage
Dominant language
C#
Stars
4.8k
Forks
4.3k
Avg merge
2d 17h
Merged PRs (30d)
51

Description

### Description

`Invoke-AzDataFactoryV2Pipeline` is being called from multiple ThreadJobs, simultaneously, for the same pipeline, with different execution parameters.

What happens sometimes is that this line
`$pipelineRunId = Invoke-AzDataFactoryV2Pipeline -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineName $PipelineName -Parameter $ExecParams`
will not invoke a new pipeline but instead returns the pipelineRunId from the just previously triggered execution, even if the parameters don't match, if the timing is just right (or wrong, in this case).
It does not throw any error at these step, only our internal validation did when it realized it was not the expected pipeline run.

I created a test ADF pipeline that just waits for the given input of seconds
![image](https://github.com/user-attachments/assets/3588f462-a4ba-4e39-b8e9-eaaac57fbf8a)

and then I used this script to reproduce the issue
```
param(
$ResourceGroupName,
$DataFactoryName,
$PipelineName,
$ConcurrentCalls
)

Get-Job | Remove-Job -Force

$scriptBlock = {
try{
$ResourceGroupName = $using:ResourceGroupName
$DataFactoryName = $using:DataFactoryName
$PipelineName = $using:PipelineName
$i = $using:i

$randomWait = (Get-Random -Maximum 5 -Minimum 3)
$ExecParams = @{ }
$ExecParams.Add("WaitTime", "$randomWait")
$ExecParams.Add("JobId", "$i")

Write-Host "START: $(get-date)"

Write-Host "ExecParams:"
$ExecParams | Out-String | Write-Host

# Invoking ADF Pipeline
$pipelineRunId = Invoke-AzDataFactoryV2Pipeline -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineName $PipelineName -Parameter $ExecParams

Write-Host "PROGRESS: pipelineRunId: $pipelineRunId"

Write-Host "DONE: $(get-date)"
}
catch {
throw
}
}

for ($i = 1; $i -le $ConcurrentCalls; $i++) {
$job = Start-ThreadJob -Name "Job for iteration $i" -ScriptBlock $scriptBlock
}

Start-Sleep 1

while ($true) {
$completedJobs = Get-Job -State Completed
$runningJobs = Get-Job -State Running
$notStartedJobs = Get-Job -State NotStarted

foreach ($job in $completedJobs) {
Write-Host "Found completed jobs:" -ForegroundColor Cyan
Receive-Job -Id $job.Id

Write-Host "Cleanup completed jobs" -ForegroundColor Cyan
Remove-Job -Id $job.Id
}

# once no more running jobs, exit
if(($runningJobs.count + $notStartedJobs.Count) -eq 0){
break
}
Start-Sleep 1
}

```
It's not consistent but, as shown below, it can return the same pipelineRunId for different jobs (and each job has a different parameter and should trigger a different pipeline execution)

As a work around, I created a function `Invoke-AzDataFactoryV2PipelineSafely` that will only "accept" the pipelineRunId if one distinctive parameter is the same from the call and from the info of the returned pipeline.

![image](https://github.com/user-attachments/assets/e9b37b5b-6150-4a91-a7c8-02e5c72a6da9)

### Issue script & Debug output

```PowerShell
& "$rootPath\InvokeAzDataFactoryV2Pipeline_repro.ps1" -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -PipelineName $PipelineName -ConcurrentCalls 3

Found completed jobs:
START: 07/23/2024 15:07:55
ExecParams:

Name Value
---- -----
WaitTime 4
JobId 1

PROGRESS: pipelineRunId: 4033f90c-2a3a-4768-a0c4-76ff399dfb0e
DONE: 07/23/2024 15:07:56
Cleanup completed jobs
Found completed jobs:
START: 07/23/2024 15:07:55
ExecParams:

Name Value
---- -----
WaitTime 4
JobId 2

PROGRESS: pipelineRunId: ac5b4fc5-d978-476e-a491-76b74846d7ab
DONE: 07/23/2024 15:07:56
Cleanup completed jobs
Found completed jobs:
START: 07/23/2024 15:07:55
ExecParams:

Name Value
---- -----
WaitTime 4
JobId 3

PROGRESS: pipelineRunId: ac5b4fc5-d978-476e-a491-76b74846d7ab
DONE: 07/23/2024 15:07:56
Cleanup completed jobs
```

### Environment data

```PowerShell
Name Value
---- -----
PSVersion 7.4.3
PSEdition Core
GitCommitId 7.4.3
OS Microsoft Windows 10.0.19045
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
```

### Module versions

```PowerShell
ModuleType Version PreRelease Name PSEdition ExportedCommands
---------- ------- ---------- ---- --------- ----------------
Script 3.0.2 Az.Accounts Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave…}
Script 2.13.1 Az.Accounts Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave…}
Script 1.1.4 Az.AnalysisServices Core,Desk {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysisServicesServer, Remove-AzAnalysisServicesServer…}
Script 1.18.5 Az.DataFactory Core,Desk {Add-AzDataFactoryV2DataFlowDebugSessionPackage, Add-AzDataFactoryV2TriggerSubscription, Get-AzDataFactory, Get-AzDataFactoryActivityWindow…}
Script 4.9.3 Az.KeyVault Core,Desk {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertificateOperation, Get-AzKeyVaultCertificateOperation…}
Script 6.5.3 Az.Resources Core,Desk {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssignment…}
Script 5.10.1 Az.Storage Core,Desk {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAccountKey…}
Script 1.3.0 azure.datafactory.tools Desk {Publish-AdfV2FromJson, Publish-AdfV2UsingArm, Get-AdfFromService, New-AdfPublishOption…}
```

### Error output

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.