Hangfire failing to create large batches... The transaction associated with the current connection has completed but has not been disposed

We are getting an exception when creating a batch.

I am pushing the batch heavily.... 40 jobs running per machine on 10 machines backed by a SQL database.

The exception happens during the save of the created batch/sub-batches/jobs. This doesn't happen when the batches are 1/2 the size... In this case we are creating on the order of 2000 inter-related.

> Hangfire.CreateBatchFailedException: An exception occurred while creating the batch. See the inner exception for details. ---> Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details. ---> System.InvalidOperationException: The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements.
at System.Data.SqlClient.TdsParser.TdsExecuteRPC(SqlCommand cmd, _SqlRPC[] rpcArray, Int32 timeout, Boolean inSchema, SqlNotificationRequest notificationRequest, TdsParserStateObject stateObj, Boolean isCommandProc, Boolean sync, TaskCompletionSource`1 completion, Int32 startRpc, Int32 startParam)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<>c__DisplayClass21_0.b__0(DbConnection connection, DbTransaction transaction)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.b__5_0(DbConnection connection, DbTransaction transaction)
at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass21_0.b__0(DbConnection connection, DbTransaction transaction)
at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](Func`2 func)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction[T](Func`3 func, Nullable`1 isolationLevel)
at Hangfire.SqlServer.SqlServerStorage.UseTransaction(Action`2 action)
at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit()
at Hangfire.Batches.AwaitingBatchContinuationSupport.OnStateElection(ElectStateContext context)
at Hangfire.States.StateMachine.ApplyState(ApplyStateContext initialContext)
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass7_0.b__0()
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
--- End of inner exception stack trace ---
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
at Javelin.MultiAsset.Batch.BatchService.<>c__DisplayClass49_2.b__10(IBatchAction batch) in Y:\TeamCity\BuildAgent3\work\15a46b9a78339ebc\Javelin.MultiAsset.Batch\BatchService.cs:line 805
at Hangfire.BatchJobClient.Create(Action`1 createAction, IBatchState state, String description)
--- End of inner exception stack trace ---
at Hangfire.BatchJobClient.Create(Action`1 createAction, IBatchState state, String description)
at Hangfire.BatchJob.AwaitBatch(String antecedentId, Action`1 createAction, String description, BatchContinuationOptions options)
at Javelin.MultiAsset.Batch.BatchService.CreatePortfolioAnalysisBatch(AnalysisBatchRequest analysisBatchRequest, IJobCancellationToken cancellationToken, PerformContext performContext) in Y:\TeamCity\BuildAgent3\work\15a46b9a78339ebc\Javelin.MultiAsset.Batch\BatchService.cs:line 759
at Javelin.MultiAsset.BackendService.HangFireEngine.CreatePortfolioAnalysisBatch(AnalysisBatchRequest analysisBatchRequest, IJobCancellationToken cancellationToken, PerformContext performContext) in Y:\TeamCity\BuildAgent3\work\15a46b9a78339ebc\Javelin.MultiAsset.BackendService\HangFireEngine.cs:line 272
>

There doesn't seem to be any extraneous load on the database, or the servers. Database memory utilization is high though.


Start with Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit, especially SetRangeInHash, and trace the transaction flow through SqlServerStorage.UseTransaction during Hangfire.BatchJobClient.Create. Reproduce the failure with a large batch of inter-related jobs against SQL Server, then verify that batch creation completes without the transaction-disposal exception.

Written by the indexing model from the issue text.