HangfireIO / HangfireIO/Hangfire
SQLException "Timeout" when calling Hangfire.RecurringJob.AddOrUpdate(...)
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
Hi!
Since the last update to
"Hangfire.Console": "1.2.0",
"Hangfire.Core": "1.6.8",
"Hangfire.Dashboard.Authorization": "2.1.0",
"Hangfire.SqlServer": "1.6.8"
We get a "SQLException: Timeout" twice a week when our IIS website wakes up again and (during its startup process) calls
`Hangfire.RecurringJob.AddOrUpdate(...)`
the full exception stacktrace is
```
[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. This failure occurred while attempting to connect to the routing destination. The duration spent while attempting to connect to the original server was - [Pre-Login] initialization=4; handshake=15; [Login] initialization=0; authentication=0; [Post-Login] complete=0; ]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) +388
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +717
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +4515
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) +6558377
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds) +6561143
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite) +586
System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) +742
System.Data.SqlClient.SqlCommand.ExecuteNonQuery() +287
Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) +1348
Hangfire.SqlServer.SqlServerWriteOnlyTransaction.b__5_0(DbConnection connection, DbTransaction transaction) +468
Hangfire.SqlServer.<>c__DisplayClass19_0.b__0(DbConnection connection, DbTransaction transaction) +15
Hangfire.SqlServer.SqlServerStorage.UseConnection(Func`2 func) +80
Hangfire.SqlServer.SqlServerStorage.UseTransaction(Func`3 func, Nullable`1 isolationLevel) +350
Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit() +197
Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options) +596
Hangfire.RecurringJob.AddOrUpdate(Expression`1 methodCall, String cronExpression, TimeZoneInfo timeZone, String queue) +275
Abp.Zero.JobScheduling.<>c__DisplayClass14_0`1.b__0() +92
Abp.Zero.JobScheduling.HangFireBackgroundJobScheduler.WrapException(Action action) +70
MyApp.Reporting.Jobs.CleanTempFolderJobRegistrar.Run() +341
MyApp.Web.Startup.Configuration(IAppBuilder app, HttpConfiguration config, IIocManager iocManager, Boolean skipConfigurators) +605
MyApp.Web.Startup.Configuration(IAppBuilder app) +73
```
I remember that I was told once it would be an issue with the connection pool size, however this is our connection string:
``
And calling sp_who on this database shows that output:
```
spid ecid status loginame hostname blk dbname cmd request_id
85 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
100 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
104 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
105 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
106 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
107 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
108 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
109 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
110 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
111 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
112 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
113 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
114 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
115 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
116 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
117 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
118 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
120 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
122 0 suspended MYDATABASE_admin MYDATABASE 112 MYDATABASE EXECUTE 0
123 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
124 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
125 0 runnable MYDATABASE_admin LAPTOP-3M6KT9XX 0 MYDATABASE SELECT 0
126 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
127 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
128 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
129 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
130 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
132 0 sleeping MYDATABASE_admin MYDATABASE 0 MYDATABASE AWAITING COMMAND 0
```
So there are only like 20 connections and most of them are sleeping.
Any idea what this could be?
The only way to overcome this issue is doing an acutal **IIS reset** (which is BAD of course)
Contributor guide
Research direction
Start at Hangfire.RecurringJob.AddOrUpdate and follow the stack trace through Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit and SqlServerStorage.UseTransaction. Compare the startup connection string settings with the SQL Server timeout and sp_who output, then determine the conditions that reproduce the timeout. Done means the cause is isolated and a verified resolution is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100