HangfireIO / HangfireIO/Hangfire

The timeout elapsed prior to obtaining a distributed lock

Open
#584 2 comments 3 reactions 0 assignees View on GitHub
t: question
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

We have multiple jobs and put DisableConcurrentExecution(600) but some operations can last more than 10 mins. In that case, we get following error:

```
2016-05-31 15:40:02,272 [Worker #8a94a848] ERROR Hangfire.AutomaticRetryAttribute - Failed to process the job '210619': an exception occurred.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:DataUploader.Start' resource.
at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
at Hangfire.SqlServer.SqlServerDistributedLock..ctor(SqlServerStorage storage, String resource, TimeSpan timeout)
at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
```

If operation lasts less than 10 mins, everything works fine.

Contributor guide

Open the contributing guide

Research direction

Start with the stack-trace entry points SqlServerDistributedLock.Acquire and DisableConcurrentExecutionAttribute.OnPerforming. Investigate how the distributed-lock timeout relates to jobs running longer than 10 minutes, and reproduce the reported failure with a long-running operation. Done means the timeout behavior is understood and the issue has a documented, tested resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.