Azure / Azure/azure-functions-powershell-worker

Wait-DurableTask broken in Powershell 7.2 functions

Open
#851 13 comments 0 reactions 0 assignees View on GitHub
feature:durable functions
Dominant language
C#
Stars
215
Forks
61
Avg merge
21h 4m
Merged PRs (30d)
6

Description

It seems there is inconsistency and breakage with external event handling when using Powershell 7.2 in Azure functions.

HttpTrigger
```powershell
using namespace System.Net

param($Request, $TriggerMetadata)

$inputs = @{
Name = "$($Request.Query.Name)"
}

$FunctionName = $Request.Params.FunctionName
$InstanceId = Start-DurableOrchestration -FunctionName $FunctionName -Input $inputs
Write-Host "Started orchestration with ID = '$InstanceId'"

$Response = New-DurableOrchestrationCheckStatusResponse -Request $Request -InstanceId $InstanceId
Push-OutputBinding -Name Response -Value $Response
```

Orchestration function
```powershell
using namespace System.Net
param($Context)

$output = @()
$gate1 = Start-DurableExternalEventListener -EventName "Paris" -NoWait -verbose
$gate2 = Start-DurableExternalEventListener -EventName "London" -NoWait -verbose

$output = Invoke-DurableActivity -FunctionName 'Hello' -Input $Context.Input

$endResults = Wait-DurableTask -Task @($gate1, $gate2)

$finaloutput = Invoke-ActivityFunction -FunctionName 'Bye' -Input $output

$finaloutput

```

Hello (Activity)
```powershell
using namespace System.Net
param($name, $TriggerMetadata)

$InstanceId = $TriggerMetadata.InstanceId

Write-Host "Hello $name"

Send-DurableExternalEvent -InstanceId $InstanceId -EventName "London" -verbose
Send-DurableExternalEvent -InstanceId $InstanceId -EventName "Paris" -verbose

$name
```

Bye (Activity)
```powershell
using namespace System.Net
param($name)

Write-Host "Bye $name"

$name
```

Using the above code when I do:

"FUNCTIONS_WORKER_RUNTIME_VERSION": "~7"

It works and will run Bye and produce the correct output. However, when I change it to:

"FUNCTIONS_WORKER_RUNTIME_VERSION": "7.2"

It stays in the running state and does not run Bye. Sending the 2 events manually again, it throws an exception below:
```powershell
Orchestration completed with a 'Failed' status and 314 bytes of output. Details: Unhandled exception while executing orchestration: DurableTask.Core.Exceptions.NonDeterministicOrchestrationException: Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence ID 1 and name 'Bye' (version ''), but the current replay execution hasn't (yet?) scheduled this task. Was a change made to the orchestrator code after this instance had already started running?
[2022-08-26T06:44:07.415Z] at DurableTask.Core.TaskOrchestrationContext.HandleTaskScheduledEvent(TaskScheduledEvent scheduledEvent) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 271
[2022-08-26T06:44:07.415Z] at DurableTask.Core.TaskOrchestrationExecutor.ProcessEvent(HistoryEvent historyEvent) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 189
[2022-08-26T06:44:07.416Z] at DurableTask.Core.TaskOrchestrationExecutor.g__ProcessEvents|11_0(IEnumerable`1 events) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 114
[2022-08-26T06:44:07.416Z] at DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore(IEnumerable`1 pastEvents, IEnumerable`1 newEvents) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 122
[2022-08-26T06:44:07.417Z] at DurableTask.Core.TaskOrchestrationContext.HandleTaskScheduledEvent(TaskScheduledEvent scheduledEvent) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 271
[2022-08-26T06:44:07.417Z] at DurableTask.Core.TaskOrchestrationExecutor.ProcessEvent(HistoryEvent historyEvent) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 189
[2022-08-26T06:44:07.420Z] at DurableTask.Core.TaskOrchestrationExecutor.g__ProcessEvents|11_0(IEnumerable`1 events) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 114
[2022-08-26T06:44:07.420Z] at DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore(IEnumerable`1 pastEvents, IEnumerable`1 newEvents) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 122
```

If I make the orchestrator slightly complicated:

Orchestration function
```powershell
using namespace System.Net
param($Context)

$output = @()
$gate1 = Start-DurableExternalEventListener -EventName "Paris" -NoWait -verbose
$gate2 = Start-DurableExternalEventListener -EventName "London" -NoWait -verbose

$output = Invoke-DurableActivity -FunctionName 'Hello' -Input $Context.Input
$output1 = Invoke-DurableActivity -FunctionName 'Hello1' -Input $output

$endResults = Wait-DurableTask -Task @($gate1, $gate2)

$finaloutput = Invoke-ActivityFunction -FunctionName 'Bye' -Input $output1

$finaloutput
```

Hello1 (Activity)
```powershell
using namespace System.Net
param($name)

Write-Host "Hello again $name!!"

$name

```

Then it again does not complete unless I resend the events, but this time I sometimes have to send the events multiple times before it fails again (using ~7 still works though). The more complex the orchestration the more inconsistent it seems to get.

Core Tools Version: 4.0.4736 Commit hash: N/A (64-bit)
Function Runtime Version: 4.8.1.18957

When using ~7 it runs PowerShell 7.0.11
When using 7.2 its running PowerShell 7.2.4

Am I doing something wrong?
Anyone else have success running external events with PowerShell 7.2 when using complex orchestrations?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the orchestration with the shown HttpTrigger, orchestration, Hello, and Bye functions under PowerShell 7.0.11 and 7.2.4, comparing FUNCTIONS_WORKER_RUNTIME_VERSION ~7 and 7.2. Inspect the behavior around Start-DurableExternalEventListener and Wait-DurableTask. Done means both external events allow the orchestration to run Bye without repeated event delivery or a non-deterministic workflow exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, powershell
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.