HangfireIO / HangfireIO/Hangfire

Hangfire.Storage.DistributedLockTimeoutException: Timeout expired.

Open
#998 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

I updated my hangfire version to use the very latest update to Pro.Redis (v2.1.2), and now, I'm getting a new symptom.

My jobs ( 4300 simultaneous jobs running on ~170 machines ) ground to a halt... for a period of time - didn't measure how long, but
Based on the one job which aborted, it must have been due to a deadlock due to this RedisDistributedLock hanging.

Hangfire.CreateBatchFailedException
An exception occurred while creating the batch. See the inner exception for details.
Hangfire.CreateBatchFailedException: An exception occurred while creating the batch. See the inner exception for details. ---> Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'batch:0a6ac48e-a0f7-4040-9195-0b6a6dcd4ffc:lock' resource.
at Hangfire.Pro.Redis.RedisDistributedLock.RetryUntilTrue(Func`1 action, Nullable`1 timeOut)
at Hangfire.Pro.Redis.RedisDistributedLock..ctor(Func`1 databaseFactory, String resource, String lockToken, TimeSpan timeout)
at Hangfire.Pro.Redis.RedisConnection.DistributedLockWrapper..ctor(RedisConnection connection, String resource, TimeSpan timeout)
at Hangfire.Pro.Redis.RedisConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.Batches.CachedStateConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.Batches.BatchJobStorageExtensions.AcquireDistributedBatchLock(IStorageConnection connection, String batchId, TimeSpan timeout)
at Hangfire.Batches.BatchContinuationsSupportAttribute.AddContinuation(ElectBatchStateContext context, BatchAwaitingState awaitingState)
at Hangfire.Batches.BatchContinuationsSupportAttribute.OnStateElection(ElectBatchStateContext context)
at Hangfire.Batches.States.BatchStateMachine.ApplyState(ApplyBatchStateContext context)
at Hangfire.Batches.Client.BatchFactory.Create(BatchCreateContext context)
at Hangfire.BatchJobClient.Create(Action`1 createAction, IBatchState state, String description)
--- End of inner exception stack trace ---
at Hangfire.BatchJobClient.Create(Action`1 createAction, IBatchState state, String description)
at Hangfire.BatchJob.AwaitBatch(String antecedentId, Action`1 createAction, String description, BatchContinuationOptions options)
at Javelin.MultiAsset.Batch.BatchService.CreatePortfolioAnalysisBatchJobs(String parentBatchId, AnalysisBatchRequest analysisBatchRequest, PerformContext performContext, IJobCancellationToken cancellationToken) in Z:\TeamCity\BuildAgent2\work\15a46b9a78339ebc\Javelin.MultiAsset.Batch\BatchService.cs:line 1292

Hangfire.Console v 1.3.6
Hangfire.Core v1.6.16
Hangfire.Pro v2.1.0
Hangfire.Pro.redis v2.1.2
Hangfire.SqlServer v1.6.16
Hangfire.Unity v1.3.0

Contributor guide

Open the contributing guide

Research direction

Start with RedisDistributedLock.RetryUntilTrue and the RedisConnection.AcquireDistributedLock path shown in the stack trace, then inspect the batch continuation flow through BatchContinuationsSupportAttribute. The reported caller is BatchService.cs line 1292. Done means the reported distributed-lock timeout can be reproduced or its cause identified and the batch operation no longer hangs under the stated workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.