HangfireIO / HangfireIO/Hangfire
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired.
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
I updated my hangfire version to use the very latest update to Pro.Redis (v2.1.2), and now, I'm getting a new symptom.
My jobs ( 4300 simultaneous jobs running on ~170 machines ) ground to a halt... for a period of time - didn't measure how long, but
Based on the one job which aborted, it must have been due to a deadlock due to this RedisDistributedLock hanging.
Hangfire.CreateBatchFailedException
An exception occurred while creating the batch. See the inner exception for details.
Hangfire.CreateBatchFailedException: An exception occurred while creating the batch. See the inner exception for details. ---> Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'batch:0a6ac48e-a0f7-4040-9195-0b6a6dcd4ffc:lock' resource.
at Hangfire.Pro.Redis.RedisDistributedLock.RetryUntilTrue(Func`1 action, Nullable`1 timeOut)
at Hangfire.Pro.Redis.RedisDistributedLock..ctor(Func`1 databaseFactory, String resource, String lockToken, TimeSpan timeout)
at Hangfire.Pro.Redis.RedisConnection.DistributedLockWrapper..ctor(RedisConnection connection, String resource, TimeSpan timeout)
at Hangfire.Pro.Redis.RedisConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.Batches.CachedStateConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.Batches.BatchJobStorageExtensions.AcquireDistributedBatchLock(IStorageConnection connection, String batchId, TimeSpan timeout)
at Hangfire.Batches.BatchContinuationsSupportAttribute.AddContinuation(ElectBatchStateContext context, BatchAwaitingState awaitingState)
at Hangfire.Batches.BatchContinuationsSupportAttribute.OnStateElection(ElectBatchStateContext context)
at Hangfire.Batches.States.BatchStateMachine.ApplyState(ApplyBatchStateContext context)
at Hangfire.Batches.Client.BatchFactory.Create(BatchCreateContext context)
at Hangfire.BatchJobClient.Create(Action`1 createAction, IBatchState state, String description)
--- End of inner exception stack trace ---
at Hangfire.BatchJobClient.Create(Action`1 createAction, IBatchState state, String description)
at Hangfire.BatchJob.AwaitBatch(String antecedentId, Action`1 createAction, String description, BatchContinuationOptions options)
at Javelin.MultiAsset.Batch.BatchService.CreatePortfolioAnalysisBatchJobs(String parentBatchId, AnalysisBatchRequest analysisBatchRequest, PerformContext performContext, IJobCancellationToken cancellationToken) in Z:\TeamCity\BuildAgent2\work\15a46b9a78339ebc\Javelin.MultiAsset.Batch\BatchService.cs:line 1292
Hangfire.Console v 1.3.6
Hangfire.Core v1.6.16
Hangfire.Pro v2.1.0
Hangfire.Pro.redis v2.1.2
Hangfire.SqlServer v1.6.16
Hangfire.Unity v1.3.0
Contributor guide
Research direction
Start with RedisDistributedLock.RetryUntilTrue and the RedisConnection.AcquireDistributedLock path shown in the stack trace, then inspect the batch continuation flow through BatchContinuationsSupportAttribute. The reported caller is BatchService.cs line 1292. Done means the reported distributed-lock timeout can be reproduced or its cause identified and the batch operation no longer hangs under the stated workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, redis
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100