HangfireIO / HangfireIO/Hangfire
The timeout elapsed prior to obtaining a distributed lock
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
We have multiple jobs and put DisableConcurrentExecution(600) but some operations can last more than 10 mins. In that case, we get following error:
```
2016-05-31 15:40:02,272 [Worker #8a94a848] ERROR Hangfire.AutomaticRetryAttribute - Failed to process the job '210619': an exception occurred.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:DataUploader.Start' resource.
at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
at Hangfire.SqlServer.SqlServerDistributedLock..ctor(SqlServerStorage storage, String resource, TimeSpan timeout)
at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
```
If operation lasts less than 10 mins, everything works fine.
Contributor guide
Research direction
Start with the stack-trace entry points SqlServerDistributedLock.Acquire and DisableConcurrentExecutionAttribute.OnPerforming. Investigate how the distributed-lock timeout relates to jobs running longer than 10 minutes, and reproduce the reported failure with a long-running operation. Done means the timeout behavior is understood and the issue has a documented, tested resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100