HangfireIO / HangfireIO/Hangfire

Hangfire Load by waits (AAS)

Open
#1,906 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

Firstly - thanks for an awesome library!

We're seeing an issue with Hangfire suddenly (in the last 24 hours) consuming a large amount of our database CPU.

The queries that seem to be taking over are:

```
delete top (1) JQ
output DELETED.Id, DELETED.JobId, DELETED.Queue
from [HangFire].JobQueue JQ with (readpast, updlock, rowlock, forceseek)
where Queue in (@queues1) and (FetchedAt is null or FetchedAt < DATEADD(second, @timeout, GETUTCDATE()))
```

And the very similar:

```
delete top (1) JQ
output DELETED.Id, DELETED.JobId, DELETED.Queue
from [HangFire].JobQueue JQ with (readpast, updlock, rowlock, forceseek)
where Queue in (@queues1,@queues2) and (FetchedAt is null or FetchedAt < DATEADD(second, @timeout, GETUTCDATE()))
```

I've upgraded from Hangfire Core and SQLServer 1.7.15 -> 1.7.24, but the issue remains.

Hangfire is running on a single server, with 4 queues and 121 "workers". Split 1/50/50/20.

Even running a select on the JobQueue table seems to take > 10 seconds. Is there a sensible check we can make to work out what's causing this table to suddenly respond so slowly?

Contributor guide

Open the contributing guide

Research direction

Start with the reported DELETE TOP (1) queries against [HangFire].JobQueue and the reported slow SELECT, comparing behavior between Hangfire 1.7.15 and 1.7.24 under the four-queue, 121-worker setup. Identify the database or workload condition causing the slowdown and document a reproducible check and resolved behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.