Retention is crashing the service at runtime.

Hello,
We’re using the UseRetention feature. It worked without issues at first, but later we started getting an error and the service keeps crashing. It seems data-related, but what I don’t understand is why it’s trying to deserialize the data of the records that are supposed to be deleted.

The error we’re getting is below:

`Unhandled exception. System.ArgumentException: An item with the same key has already been added. Key: Outcomes
at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at Elsa.Workflows.Serialization.Converters.PolymorphicDictionaryConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Deserialize(Utf8JsonReader& reader, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo`1 jsonTypeInfo, Nullable`1 actualByteCount)
at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo`1 jsonTypeInfo)
at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
at Elsa.Workflows.Serialization.Serializers.JsonPayloadSerializer.Deserialize[T](String payload)
at Elsa.EntityFrameworkCore.Modules.Runtime.EFCoreActivityExecutionStore.DeserializePayload[T](RuntimeElsaDbContext dbContext, ActivityExecutionRecord entity, String propertyName)
at Elsa.EntityFrameworkCore.Modules.Runtime.EFCoreActivityExecutionStore.OnLoadAsync(RuntimeElsaDbContext dbContext, ActivityExecutionRecord entity, CancellationToken cancellationToken)
at Elsa.EntityFrameworkCore.Store`2.QueryAsync(Func`2 query, Func`4 onLoading, Boolean ignoreQueryFilters, CancellationToken cancellationToken)
at Elsa.EntityFrameworkCore.Store`2.QueryAsync(Func`2 query, Func`4 onLoading, Boolean ignoreQueryFilters, CancellationToken cancellationToken)
at Elsa.EntityFrameworkCore.Store`2.QueryAsync(Func`2 query, Func`4 onLoading, CancellationToken cancellationToken)
at Open.Linq.AsyncExtensions.Extensions.ToList[TSource](Task`1 source)
at Elsa.EntityFrameworkCore.Modules.Runtime.EFCoreActivityExecutionStore.FindManyAsync(ActivityExecutionRecordFilter filter, CancellationToken cancellationToken)
at Elsa.Retention.Collectors.ActivityExecutionRecordCollector.GetRelatedEntities(ICollection`1 workflowInstances)+MoveNext()
at Elsa.Retention.Collectors.ActivityExecutionRecordCollector.GetRelatedEntities(ICollection`1 workflowInstances)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()
at Elsa.Retention.Contracts.IRelatedEntityCollector`1.Elsa.Retention.Contracts.IRelatedEntityCollector.GetRelatedEntitiesGeneric(ICollection`1 workflowInstances)+MoveNext()
at Elsa.Retention.Contracts.IRelatedEntityCollector`1.Elsa.Retention.Contracts.IRelatedEntityCollector.GetRelatedEntitiesGeneric(ICollection`1 workflowInstances)+MoveNext()
at Elsa.Retention.Contracts.IRelatedEntityCollector`1.Elsa.Retention.Contracts.IRelatedEntityCollector.GetRelatedEntitiesGeneric(ICollection`1 workflowInstances)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()
at Elsa.Retention.Jobs.CleanupJob.ExecuteAsync(CancellationToken cancellationToken)
at Elsa.Retention.Jobs.CleanupJob.ExecuteAsync(CancellationToken cancellationToken)
at Elsa.Retention.CleanupRecurringTask.ExecuteAsync(CancellationToken stoppingToken)
at Elsa.Common.Multitenancy.TaskExecutor.ExecuteInternalAsync(ITask task, Func`1 action, CancellationToken cancellationToken)
at Elsa.Common.Multitenancy.TaskExecutor.ExecuteInternalAsync(ITask task, Func`1 action, CancellationToken cancellationToken)
at Elsa.Common.Multitenancy.TaskExecutor.ExecuteTaskAsync(ITask task, CancellationToken cancellationToken)
at Elsa.Common.Multitenancy.EventHandlers.StartRecurringTasks.<>c__DisplayClass5_1.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Elsa.Common.RecurringTasks.ScheduledTimer.Callback(Object state)
at System.Threading.Tasks.Task.<>c.b__128_1(Object state)
at System.Threading.QueueUserWorkItemCallback.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()`

Do you have any guidance? What kind of data-related issue might be causing this? @sfmskywalker

Updated:

Some ActivityExecutionRecords items has 2 more outcomes. mistake data perhaps but crash servis to.