SqlException Transaction (Process ID 66) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

Hi all
We are using durable functions in a stand alone process (not azure functions) in kubernetes cluster.
Microsoft.DurableTask.SqlServer.AzureFunctions nuget version 1.1.1

Sometimes when calling ReadEntityStateAsync we get an exception:

Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

It happens once every few thousands of orchestrations.

The full call stack:

Microsoft.VideoIndexer.Telemetry.ViException: Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at Microsoft.Data.SqlClient.SqlDataReader.Read()
   at DurableTask.SqlServer.SqlOrchestrationService.GetOrchestrationStateAsync(String instanceId, String executionId, CancellationToken cancellationToken) in /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 614
   at DurableTask.SqlServer.AzureFunctions.SqlDurabilityProvider.RetrieveSerializedEntityState(EntityId entityId, JsonSerializerSettings serializierSettings) in /_/src/DurableTask.SqlServer.AzureFunctions/SqlDurabilityProvider.cs:line 103
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.ReadEntityStateAsync[T](DurabilityProvider provider, EntityId entityId) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableClient.cs:line 566
   at Microsoft.VideoIndexer.IndexingService.Common.ActivityRunner.RunActivity[TContext](Func`3 function, TContext context, IDurableEntityClient entityClient, Boolean throwExceptions, String callingMethod) in C:\__w\1\s\Src\Microsoft.VideoIndexer.IndexingService.Common\ActivityRunner.cs:line 121
ClientConnectionId:dba86c5b-4d3b-4f29-89b9-d169ac0d9362
Error Number:1205,State:51,Class:13
   --- End of inner exception stack trace ---

Start with GetOrchestrationStateAsync in src/DurableTask.SqlServer/SqlOrchestrationService.cs at line 614, then trace its call from RetrieveSerializedEntityState in src/DurableTask.SqlServer.AzureFunctions/SqlDurabilityProvider.cs at line 103. Investigate the SQL Server error 1205 during ReadEntityStateAsync and determine a tested way to prevent or handle the deadlock; done means the reported failure is addressed without regressing entity-state reads.