HangfireIO / HangfireIO/Hangfire
Next execution in the past if one server is stopped
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
If I have only one instance running then it's fine but in cases like:
- Azure scaling instance count up and then scaling down
- Running an app on a dev server and at the same time running one instance locally (connected to dev hangfire database) and then closing the local instance
Jobs don't start anymore. Next execution time comes but the job doesn't get queued and since then the time is not refreshed anymore and gets stuck in the past.
I'm using 1.7.33.
This is how it's configured
```
services.AddHangfire(configuration => configuration
.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseConsole()
.UseMAMQSqlServerStorage(Configuration.GetConnectionString("HangfireConnection"), new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
UseRecommendedIsolationLevel = true,
DisableGlobalLocks = true,
SchemaName = "appName"
},
new[] { HangfireHelper.QueueName }));
services.AddHangfireServer(options =>
{
options.Queues = new[] { HangfireHelper.QueueName };
});
```
QueueName is "queue-{machine name}"
And jobs are added like this
```
var queueName = HangfireHelper.QueueName;
JobStorage.Current = new MAMQSqlServerStorage(Configuration.GetConnectionString("HangfireConnection"), new SqlServerStorageOptions()
{
SchemaName = HangfireHelper.Schema
}, new[] { queueName });
using (var connection = JobStorage.Current.GetConnection())
{
foreach (var recurringJob in connection.GetRecurringJobs())
{
if (recurringJob.Queue == queueName)
{
RecurringJob.RemoveIfExists(recurringJob.Id);
}
}
}
/////
RecurringJob.AddOrUpdate("JobName", () => jobService.DoJobAsync(param), appSettings.HangfireJobs.JobName, queue: queueName);
```
To me it looks like the newest instance becomes a "main" instance and when it gets shut down, the existing instance doesn't take over.
My desired outcome is if there are multiple servers the job should execute only on one server, not all of them (doesn't matter which one) and when I shut down some instance it shouldn't stop the jobs from getting queued, other instances should still be able to pick them up.
Contributor guide
Research direction
No files or tests are identified. Start by reproducing the shared-database setup with multiple servers, then inspect how recurring jobs are scheduled when one server stops; done means another running server queues the job and it executes only once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, sql
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100