HangfireIO / HangfireIO/Hangfire

A solution to solve the RecurringJobScheduler trigger a job twice in sometimes when use multiple BackgroundJobServers

Open
#1,208 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

Hello Hangfire Team,

`hangfire version: v1.7.0-betal`
`BackgroundJobServers: 2`

Sometimes, a job will be triggered twice, and the interval between the two jobs within 1 sec.
The [RecurringJobScheduler](https://github.com/HangfireIO/Hangfire/blob/v1.7.0-beta1/src/Hangfire.Core/Server/RecurringJobScheduler.cs) trigger a job depend on it's `LastExecution` time, after check the code, and find in some case, the `LastExecution` will not be saved as expected.

* If a job was triggered successfully, then `SetRangeInHash` failed, result in the `LastExecution` is not as expected.
And the job will be triggered later.
* If a job was triggered successfully, then will set `LastExecution` to `nowInstant`. But, the `nowInstant` discarded the second, in below case the job may be tiggered twice:

``` c#
LastExecution = 03:00:00 (HH:mm:ss)
now = 04:00:05
nowInstant = 04:00:00
nextInstant = 04:00:00 // cronExpression.GetNextOccurrence(LastExecution)
nextInstant <= nowInstant, job will be triggered

after update job hash, set LastExecution = nowInstant

LastExecution = 04:00:00
now = 04:00:06
nowInstant = 04:00:00
nextInstant = 04:00:00 // cronExpression.GetNextOccurrence(LastExecution)
nextInstant <= nowInstant, job will be triggered

of course, above times should with millseconds, but in some case, it will be happen
```

So, I changed some codes to solve it, and it works for me.

``` c#
if (nextInstant <= nowInstant)
{
var state = new EnqueuedState { Reason = "Triggered by recurring job scheduler" };
if (recurringJob.ContainsKey("Queue") && !String.IsNullOrEmpty(recurringJob["Queue"]))
{
state.Queue = recurringJob["Queue"];
}

var context = new CreateContext(storage, connection, job, state);
context.Parameters["RecurringJobId"] = recurringJobId;

// my job don't work in minutely, so add one minute
changedFields.Add("LastExecution", JobHelper.SerializeDateTime(nowInstant.AddMinutes(1)));
// save the LastExecution time before trigger the job, if save failed, an exception will be throw
connection.SetRangeInHash($"recurring-job:{recurringJobId}", changedFields);

var backgroundJob = _factory.Create(context);
var jobId = backgroundJob?.Id;

if (String.IsNullOrEmpty(jobId))
{
Logger.Debug($"Recurring job '{recurringJobId}' execution at '{nowInstant}' has been canceled.");
}

// changedFields.Add("LastExecution", JobHelper.SerializeDateTime(nowInstant));
changedFields.Remove("LastExecution");
changedFields.Add("LastJobId", jobId ?? String.Empty);

nextInstant = cronExpression.GetNextOccurrence(nowInstant, timeZone);
}
```
Thanks

Contributor guide

Open the contributing guide

Research direction

Start with src/Hangfire.Core/Server/RecurringJobScheduler.cs, especially the LastExecution update and SetRangeInHash call described in the issue. Reproduce the recurring job with two BackgroundJobServers and verify behavior around sub-second scheduling and persistence failures. Done means a due recurring job is not triggered twice unexpectedly and its execution state remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.