HangfireIO / HangfireIO/Hangfire
A solution to solve the RecurringJobScheduler trigger a job twice in sometimes when use multiple BackgroundJobServers
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
Hello Hangfire Team,
`hangfire version: v1.7.0-betal`
`BackgroundJobServers: 2`
Sometimes, a job will be triggered twice, and the interval between the two jobs within 1 sec.
The [RecurringJobScheduler](https://github.com/HangfireIO/Hangfire/blob/v1.7.0-beta1/src/Hangfire.Core/Server/RecurringJobScheduler.cs) trigger a job depend on it's `LastExecution` time, after check the code, and find in some case, the `LastExecution` will not be saved as expected.
* If a job was triggered successfully, then `SetRangeInHash` failed, result in the `LastExecution` is not as expected.
And the job will be triggered later.
* If a job was triggered successfully, then will set `LastExecution` to `nowInstant`. But, the `nowInstant` discarded the second, in below case the job may be tiggered twice:
``` c#
LastExecution = 03:00:00 (HH:mm:ss)
now = 04:00:05
nowInstant = 04:00:00
nextInstant = 04:00:00 // cronExpression.GetNextOccurrence(LastExecution)
nextInstant <= nowInstant, job will be triggered
after update job hash, set LastExecution = nowInstant
LastExecution = 04:00:00
now = 04:00:06
nowInstant = 04:00:00
nextInstant = 04:00:00 // cronExpression.GetNextOccurrence(LastExecution)
nextInstant <= nowInstant, job will be triggered
of course, above times should with millseconds, but in some case, it will be happen
```
So, I changed some codes to solve it, and it works for me.
``` c#
if (nextInstant <= nowInstant)
{
var state = new EnqueuedState { Reason = "Triggered by recurring job scheduler" };
if (recurringJob.ContainsKey("Queue") && !String.IsNullOrEmpty(recurringJob["Queue"]))
{
state.Queue = recurringJob["Queue"];
}
var context = new CreateContext(storage, connection, job, state);
context.Parameters["RecurringJobId"] = recurringJobId;
// my job don't work in minutely, so add one minute
changedFields.Add("LastExecution", JobHelper.SerializeDateTime(nowInstant.AddMinutes(1)));
// save the LastExecution time before trigger the job, if save failed, an exception will be throw
connection.SetRangeInHash($"recurring-job:{recurringJobId}", changedFields);
var backgroundJob = _factory.Create(context);
var jobId = backgroundJob?.Id;
if (String.IsNullOrEmpty(jobId))
{
Logger.Debug($"Recurring job '{recurringJobId}' execution at '{nowInstant}' has been canceled.");
}
// changedFields.Add("LastExecution", JobHelper.SerializeDateTime(nowInstant));
changedFields.Remove("LastExecution");
changedFields.Add("LastJobId", jobId ?? String.Empty);
nextInstant = cronExpression.GetNextOccurrence(nowInstant, timeZone);
}
```
Thanks
Contributor guide
Research direction
Start with src/Hangfire.Core/Server/RecurringJobScheduler.cs, especially the LastExecution update and SetRangeInHash call described in the issue. Reproduce the recurring job with two BackgroundJobServers and verify behavior around sub-second scheduling and persistence failures. Done means a due recurring job is not triggered twice unexpectedly and its execution state remains consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100