elsa-workflows / elsa-workflows/elsa-core
RestartInterruptedWorkflowsTask does not restart workflows created by Elsa 3.3.5 when updating from 3.3.5 to 3.4.0
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
## Description
The issue occurs when upgrading Elsa Workflows from 3.3.5 to 3.4.0, using Elsa.MongoDb for storing workflow instances.
In 3.3.5, because controlled commit states were not implemented yet, we implemented a workaround to persist the state of the workflow after each activity was completed. We created a behavior that persists the workflow state to MongoDb, and attached it to the activities in our workflows. The code for this behavior is below:
```csharp
public class CommitWorkflowStateBehavior(IActivity owner) : Behavior(owner)
{
protected override async ValueTask ExecuteAsync(ActivityExecutionContext context)
{
var workflowStateExtractor = context.GetRequiredService();
var commitStateHandler = context.GetRequiredService();
var workflowState = workflowStateExtractor.Extract(context.WorkflowExecutionContext);
await commitStateHandler.CommitAsync(context.WorkflowExecutionContext, workflowState);
}
}
```
In 3.3.5 we also implemented our own background service for restarting interrupted workflows (as a workaround for [issue 4832](https://github.com/elsa-workflows/elsa-core/issues/4832)), which restarted workflows with status `Running` and substatus `Executing` on service start-up.
We would like to migrate to Elsa 3.4.0 and replace our custom background service with `RestartInterruptedWorkflowsTask` from 3.4.0. The problem we encountered is that `RestartInterruptedWorkflowsTask` only resumes workflows that have property `IsExecuting = true`. Because property `IsExecuting` was introduced in 3.4.0, workflows that were created in 3.3.5 and persisted by the custom behavior described above do not have this property. So they are not resumed on service startup with 3.4.0 even if they are not finished and were interrupted by the deployment going from 3.3.5 to 3.4.0.
## Steps to Reproduce
To help us identify the issue more quickly, please follow these guidelines:
1. **Detailed Steps**: Provide a step-by-step description of what leads to the bug. Be as specific as possible.
- Have an application that uses Elsa Workflows 3.3.5, with MongoDB for workflow persistence.
- Create a custom behavior that persists the state of the workflow to MongoDB.
- Create a workflow and attach the custom behavior to each of its activities. The state of the workflow instances will be persisted after each activity, instead of only at the end of the workflow.
- Start a workflow instance and interrupt the service while the workflow instance is still executing.
- Upgrade the service to use Elsa Workflows 3.4.0 and restart it.
- We were expecting `RestartInterruptedWorkflowsTask` to restart the interrupted workflows (that have status `Running` and substatus `Executing`), but it actually only resumes workflows with `IsExecuting = true`, which workflows created by Elsa version 3.3.5 do not have.
2. **Attachments**:
- **Sample Project**: I created a sample project ([Elsa340IsExecutingBug.zip](https://github.com/user-attachments/files/20710388/Elsa340IsExecutingBug.zip)) that reproduces the issue. It consists of two console apps, one using Elsa 3.3.5 and one using 3.4.0. The apps require a local instance of MongoDB (or a Docker container). Run the 3.3.5 app to create and run a new workflow. The workflow uses the `CommitWorkflowStateBehavior` described above for each of its activities. The console app will finish before the workflow is completed, so the workflow will be interrupted midway and remain persisted in MongoDb with status `Running` and substatus `Executing`. Run the 3.4.0, which will run `RestartInterruptedWorkflowsTask`. Observe in the application's console logs that the workflow is not restarted.
## Expected Behavior
I expected interrupted workflows with status `Running` and substatus `Executing` created by Elsa 3.3.5 to be resumed by `RestartInterruptedWorkflowsTask` in Elsa 3.4.0.
## Actual Behavior
`RestartInterruptedWorkflowsTask` requires the `IsExecuting` property, which is newly introduced in 3.4.0, to be present and to have value `true` to restart a workflow.
## Environment
- **Elsa Package Version**: When upgrading from 3.3.5 to 3.4.0.
## Troubleshooting Attempts
A possible solution could be to have some sort of one-time task that queries the database for workflows in status `Running` and substatus `Executing` and sets `IsExecuting` to `true`, like in the code snippet below.
But I'm not sure if this would work in a distributed scenario. Consider the following:
- We have a service that uses Elsa 3.3.5 and runs on 2 instances.
- We start deploying an upgrade to 3.4.0. So at first we have one instance running 3.4.0 and another instance running 3.3.5.
- The instance running 3.4.0 tries to set `IsExecuting = true` for all `Running - Executing` workflows. What happens if some of the workflows it tries to update were not the ones interrupted by the deployment, but workflows that are actively being executed on the instance still running 3.3.5? Would we need to use distributed locking when doing the update, or would setting `IsExecuting = true` not affect the execution of the workflow?
```csharp
var interruptedWorkflows = await workflowInstanceStore.FindManyAsync(new WorkflowInstanceFilter
{
WorkflowStatus = WorkflowStatus.Running,
WorkflowSubStatus = WorkflowSubStatus.Executing
}, stoppingToken);
foreach (var workflow in interruptedWorkflows)
{
if (!workflow.IsExecuting)
workflow.IsExecuting = true;
}
await workflowInstanceStore.SaveManyAsync(interruptedWorkflows, stoppingToken);
```
Contributor guide
Research direction
Start by inspecting RestartInterruptedWorkflowsTask and the workflow instance store behavior described in the issue. Run the attached Elsa340IsExecutingBug sample with MongoDB to reproduce the upgrade scenario. Done means interrupted 3.3.5 workflows with Running and Executing status are resumed safely during a mixed-version deployment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, mongodb
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100