HangfireIO / HangfireIO/Hangfire

Some job state rows for Processing have earlier CreatedAt than rows for Enqueued

Open
#1,073 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

We are seeing that on some jobs, the Processing state shows as occurring before the Enqueued state. The dashboard shows a negative time difference between the two states, and querying the rows in the Hangfire.State table, the Processing state row is in fact earlier than the Enqueued state row.

We are running two web servers against a single Hangfire database, so it seems like a time difference between the two servers could account for the issue. Hangfire.SqlServer appears to use the ASP.NET time rather than the database's time. One piece I can't explain though is that sometimes we're seeing this with jobs processed by web server #1 and other times it's web server #2.

See the negative time difference for Processing:
![image](https://user-images.githubusercontent.com/25795617/33854175-bb196d9e-de8e-11e7-9ad9-5666f728d95b.png)

Hangfire.Core and Hangfire.SqlServer version 1.6.17

The recurring job has the following attributes:
```
[DisableConcurrentExecution(30)]
[AutomaticRetry(Attempts = 0, OnAttemptsExceeded = AttemptsExceededAction.Delete)]
```

The source code for adding the job:
`RecurringJob.AddOrUpdate(JobName, svc => svc.FetchNewImportJobsAsync(CancellationToken.None), "*/1 * * * *", queue: "recurring");`

The recurring queue is setup as follows:
```
var serverOptions = new BackgroundJobServerOptions
{
Queues = new [] { "recurring" },
WorkerCount = 1
};

app.UseHangfireServer(serverOptions);
```

Contributor guide

Open the contributing guide

Research direction

Start with the Hangfire.SqlServer behavior that records CreatedAt values in the Hangfire.State table, using the reported two-server setup as the reproduction context. Compare the timestamps produced by the recurring job entry point, RecurringJob.AddOrUpdate, and the server setup using app.UseHangfireServer; done means Processing no longer precedes Enqueued and the dashboard shows a non-negative difference.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.