elsa-workflows / elsa-workflows/elsa-core
Inconsistency in bookmarks vs workflow instance due to non-atomic operations inside DefaultCommitStateHandler.CommitAsync
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
## Description
`DefaultCommitStateHandler.CommitAsync` persists workflow state as a sequence of independent, non-transactional await calls, with bookmark persistence first, and the workflow instance (state blob) saved last. If any operation between them fails (e.g. a SQL command timeout under load), the bookmark changes — including the removal of an auto-burned bookmark — are already committed, but the instance state save never happens. This leaves the relational bookmark store and the persisted instance state permanently divergent: the bookmark row is gone, while the (stale) instance blob still shows the activity as armed/suspended.
Because the event dispatcher resolves incoming stimuli against the bookmark store, the missing row means the activity can never be resumed again. The instance stays Running/Suspended forever with no way to recover through normal event delivery.
The relevant method is in `src/modules/Elsa.Workflows.Runtime/.../DefaultCommitStateHandler.cs`
```
public async Task CommitAsync(WorkflowExecutionContext workflowExecutionContext, WorkflowState workflowState, CancellationToken cancellationToken = default)
{
var updateBookmarksRequest = new UpdateBookmarksRequest(workflowExecutionContext, workflowExecutionContext.BookmarksDiff, workflowExecutionContext.CorrelationId);
await bookmarkPersister.PersistBookmarksAsync(updateBookmarksRequest); // (1) bookmark removal committed here
await activityExecutionLogRecordSink.PersistExecutionLogsAsync(workflowExecutionContext, cancellationToken); // (2)
await workflowExecutionLogRecordSink.PersistExecutionLogsAsync(workflowExecutionContext, cancellationToken); // (3)
await variablePersistenceManager.SaveVariablesAsync(workflowExecutionContext); // (4)
var workflowInstance = await workflowInstanceManager.SaveAsync(workflowState, cancellationToken); // (5) state blob saved last
workflowExecutionContext.ExecutionLog.Clear();
workflowExecutionContext.ClearCompletedActivityExecutionContexts();
await workflowExecutionContext.ExecuteDeferredTasksAsync();
await mediator.SendAsync(new WorkflowStateCommitted(workflowExecutionContext, workflowState, workflowInstance), cancellationToken);
}
```
**As there is no transaction / unit-of-work wrapping ops 1-to-5, a failure after the first line and before line 5 commits the bookmark mutation without the corresponding state advance.**
The auto-burn that feeds the removal into BookmarksDiff happens earlier, in DefaultActivityInvokerMiddleware.InvokeAsync (in-memory), and is only persisted at step (1) above:
```
var resumedBookmark = workflowExecutionContext.ResumedBookmarkContext?.Bookmark;
if (resumedBookmark is { AutoBurn: true })
workflowExecutionContext.Bookmarks.Remove(resumedBookmark);
```
## Steps to Reproduce
Have Elsa with EFCore on SQL Server persistence (separate runtime and management stores), distributed runtime enabled, with single or multiple instances.
Run a long-running workflow that suspends on an auto-burn bookmark - any blocking activity with AutoBurn = true, e.g.(in my case) an event-wait activity.
While a resume is being committed, induce a failure on the persistence path after the bookmark write — most easily a SQL command timeout (the writes at steps 2–5 seem to produce this the easiest as the execution-log can be heavy for large workflows). A transient DB slowdown or connection-pool exhaustion under load gives me the reproduction.
The bookmark row is deleted from the bookmark store, but the workflow instance state was not updated (the save at step 5 did not run).
Send the event the workflow was waiting on. It will not resume — there is no bookmark row to match — and the instance remains Running indefinitely.
## Expected Behavior
Bookmark persistence and workflow-instance state persistence within a single `CommitAsync` should be **atomic**. A failure while persisting any part of a committed execution step should roll back the whole step — including the bookmark removal — so the bookmark store and the instance state never diverge. On failure, the resume should remain retriable with the bookmark intact.
## Actual Behavior
The bookmark removal is committed independently and first; a failure on a later write in the same CommitAsync leaves the bookmark deleted and the instance state unchanged. The instance is permanently unresumable via event delivery and stays Running. The workflow execution log shows the activity's last event as Suspended, never Resumed, because the Resumed log entry is only flushed at steps (2)/(3) and is lost when the commit fails — **making the failure hard to diagnose from the journal alone**.
## Screenshots
None, this is not UI related.
## Environment
- Elsa Package Version: 3.6.1 & 3.6.2 :
- Scenario is OS-independent.
## Log Output
The triggering failure is a SQL command timeout on the persistence path, e.g.:
```
System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(...)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute...
```
(258 = Windows WAIT_TIMEOUT.)
Pretty much any failure in the CommitAsync produces this; a timeout on a heavy log instance writes is the most common trigger. Note the instance's own execution log will not contain a Resumed record for the affected activity, so the integrity loss is silent in the journal.
## Troubleshooting Attempts
1. Confirmed via the detection query that a set of Running instances had fewer live bookmarks than their definition requires (specifically zero auto-burn event-wait bookmarks where two were expected).
2. Decompressed the instance Data blob (Zstd) for affected instances and confirmed the bookmark is still present in the serialized state, while absent from the bookmark table — establishing table/blob divergence rather than a workflow-design issue.
3. Verified from the workflow execution log that the affected activities' last recorded event is Suspended, never Resumed — consistent with a failure early in CommitAsync (after the bookmark write at step 1, before the log flush at steps 2/3).
4. Traced the mechanism through 3.6.1 source: in-memory auto-burn in DefaultActivityInvokerMiddleware → WorkflowExecutionContext.CommitAsync() → DefaultCommitStateHandler.CommitAsync, where bookmark persistence (step 1) and instance save (step 5) are separate, unwrapped awaits.
## Additional Context
Intermittend reproduction rate, we only saw this after a very heavy load, while running with limited DTUs - this caused a cascade of DB write misses (IO Timeouts
## Related Issues
#5961 — Pretty sure this is related: Workflow instance state persistence timeout after bookmark resumption leaves instance state showing suspended, yet further activities have executed. (Same observable failure; this report also adds the root-cause mechanism in DefaultCommitStateHandler.)
#7397 — Workflows sometimes don't resume after child/sub-workflows finish in distributed environments. (Different code path, bookmark queue purge race, but similar theme: bookmark deletion not atomic with the operation that depends on it.)
Contributor guide
Assessment
This issue has not been assessed yet.