elsa-workflows / elsa-workflows/elsa-core
Workflow stuck at Pending state when unhandled exception happen on FlowChartSerialization
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
## Description
When error occured at serialization process(I Think there is a same behaviour for other unhandled exceptions). For my case it is OutOfMemoryException. Workflow stucks at pending state.
OutOfMemoryException stacktrace:
`at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags) at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 value, Int32 firstEscapeIndexVal) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at Elsa.Workflows.Serialization.Converters.PolymorphicObjectConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) at Elsa.Workflows.Serialization.Converters.PolymorphicDictionaryConverter.Write(Utf8JsonWriter writer, IDictionary`2 value, JsonSerializerOptions options) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at System.Text.Json.JsonSerializer.WriteStringAsObject(Object value, JsonTypeInfo jsonTypeInfo) at Elsa.Workflows.Serialization.Converters.PolymorphicObjectConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) at Elsa.Workflows.Serialization.Converters.PolymorphicDictionaryConverter.Write(Utf8JsonWriter writer, IDictionary`2 value, JsonSerializerOptions options) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at System.Text.Json.JsonSerializer.WriteString[TValue](TValue& value, JsonTypeInfo`1 jsonTypeInfo) at Elsa.EntityFrameworkCore.Modules.Management.EFCoreWorkflowInstanceStore.OnSaveAsync(ManagementElsaDbContext managementElsaDbContext, WorkflowInstance entity, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Store`2.SaveAsync(TEntity entity, Expression`1 keySelector, Func`4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Store`2.SaveAsync(TEntity entity, Expression`1 keySelector, Func`4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Store`2.SaveAsync(TEntity entity, Expression`1 keySelector, Func`4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.EntityStore`2.SaveAsync(TEntity entity, Func`4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Modules.Management.EFCoreWorkflowInstanceStore.SaveAsync(WorkflowInstance instance, CancellationToken cancellationToken) at Elsa.Workflows.Management.Services.WorkflowInstanceManager.SaveAsync(WorkflowInstance workflowInstance, CancellationToken cancellationToken) at Elsa.Workflows.Management.Services.WorkflowInstanceManager.SaveAsync(WorkflowState workflowState, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.DefaultCommitStateHandler.CommitAsync(WorkflowExecutionContext workflowExecutionContext, WorkflowState workflowState, CancellationToken cancellationToken) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowExecutionContext workflowExecutionContext) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowGraph workflowGraph, WorkflowState workflowState, RunWorkflowOptions options, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(WorkflowInstance workflowInstance, RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.<>c__DisplayClass8_0.<b__0>d.MoveNext() --- End of stack trace from previous location --- at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func`1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func`1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.BookmarkResumer.ResumeAsync(BookmarkFilter filter, ResumeBookmarkOptions options, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.StoreBookmarkQueue.EnqueueAsync(NewBookmarkQueueItem item, CancellationToken cancellationToken) at Paragon.LowCode.Workflows.Elsa3.ElsaActivities.Handlers.ResumeDispatchWorkflowActivity.HandleAsync(WorkflowExecuted notification, CancellationToken cancellationToken) in /_/src/Paragon.LowCode.Workflows.Elsa3.ElsaActivities/Handlers/ResumeDispatchWorkflowActivity.cs:line 74 at Elsa.Mediator.PublishingStrategies.SequentialProcessingStrategy.PublishAsync(NotificationStrategyContext context) at Elsa.Mediator.Middleware.Notification.Components.NotificationHandlerInvokerMiddleware.InvokeAsync(NotificationContext context) at Elsa.Mediator.Middleware.Notification.NotificationPipeline.ExecuteAsync(NotificationContext context) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, IEventPublishingStrategy strategy, CancellationToken cancellationToken) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, CancellationToken cancellationToken) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowExecutionContext workflowExecutionContext) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowGraph workflowGraph, WorkflowState workflowState, RunWorkflowOptions options, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(WorkflowInstance workflowInstance, RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.<>c__DisplayClass8_0.<b__0>d.MoveNext() --- End of stack trace from previous location --- at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func`1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func`1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.MassTransit.Consumers.DispatchWorkflowRequestConsumer.DispatchExistingWorkflowInstanceAsync(DispatchWorkflowDefinition message, CancellationToken cancellationToken) at Elsa.MassTransit.Consumers.DispatchWorkflowRequestConsumer.Consume(ConsumeContext`1 context) at MassTransit.DependencyInjection.ScopeConsumerFactory`1.Send[TMessage](ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/DependencyInjection/DependencyInjection/ScopeConsumerFactory.cs:line 22 at MassTransit.DependencyInjection.ScopeConsumerFactory`1.Send[TMessage](ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/DependencyInjection/DependencyInjection/ScopeConsumerFactory.cs:line 22 at MassTransit.Middleware.ConsumerMessageFilter`2.MassTransit.IFilter>.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ConsumerMessageFilter.cs:line 48 at MassTransit.Middleware.ConsumerMessageFilter`2.MassTransit.IFilter>.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ConsumerMessageFilter.cs:line 73 at Elsa.MassTransit.Middleware.TenantConsumeMiddleware`1.Send(ConsumeContext`1 context, IPipe`1 next) at Elsa.MassTransit.Middleware.TenantConsumeMiddleware`1.Send(ConsumeContext`1 context, IPipe`1 next) at MassTransit.Middleware.ScopedConsumeFilter`2.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.ScopedConsumeFilter`2.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.ScopedConsumeFilter`2.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.ScopedConsumeFilter`2.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.InMemoryOutboxFilter`2.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/InMemoryOutboxFilter.cs:line 35 at MassTransit.Middleware.InMemoryOutboxFilter`2.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/InMemoryOutboxFilter.cs:line 45 at MassTransit.Middleware.TeeFilter`1.<>c__DisplayClass5_0.<g__SendAsync|1>d.MoveNext() in /_/src/MassTransit/Middleware/TeeFilter.cs:line 40 --- End of stack trace from previous location --- at MassTransit.Middleware.ConsumeContextOutputMessageTypeFilter`1.SendToOutput(IPipe`1 next, ConsumeContext`1 pipeContext) in /_/src/MassTransit/Middleware/ConsumeContextOutputMessageTypeFilter.cs:line 76 at MassTransit.Middleware.ConsumeContextOutputMessageTypeFilter`1.SendToOutput(IPipe`1 next, ConsumeContext`1 pipeContext) in /_/src/MassTransit/Middleware/ConsumeContextOutputMessageTypeFilter.cs:line 108 at MassTransit.Middleware.ConsumeContextMessageTypeFilter.<>c__DisplayClass8_0.<g__SendAsync|0>d.MoveNext() in /_/src/MassTransit/Middleware/MessageTypeFilter.cs:line 76 --- End of stack trace from previous location --- at MassTransit.Middleware.DeserializeFilter.Send(ReceiveContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/DeserializeFilter.cs:line 40 at MassTransit.Middleware.RescueFilter`2.MassTransit.IFilter.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/RescueFilter.cs:line 43`
## Steps to Reproduce
I can't post my workflow here, because it's realy huge with customized activities.
Best way to repoduce issue is simulate unhandled exception at FlowchartJsonConverter while saving FlowChart at WorkflowInstance startup
6. **Additional Configuration**:
- MassTransit is used for jobs queue
## Expected Behavior
Workflow goes to Failed state
## Actual Behavior
Workflow stuck at Pending state
## Screenshots
If possible, add screenshots or screen recordings to help explain the problem.
## Environment
- Elsa 3.4
- Ubuntu 22.04.
## Related Issues
Link to any related issues here.
Contributor guide
Assessment
This issue has not been assessed yet.