HangfireIO / HangfireIO/Hangfire

Next execution in the past if one server is stopped

Open
#2,218 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

If I have only one instance running then it's fine but in cases like:
- Azure scaling instance count up and then scaling down
- Running an app on a dev server and at the same time running one instance locally (connected to dev hangfire database) and then closing the local instance

Jobs don't start anymore. Next execution time comes but the job doesn't get queued and since then the time is not refreshed anymore and gets stuck in the past.

I'm using 1.7.33.

This is how it's configured

```
services.AddHangfire(configuration => configuration
.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseConsole()
.UseMAMQSqlServerStorage(Configuration.GetConnectionString("HangfireConnection"), new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
UseRecommendedIsolationLevel = true,
DisableGlobalLocks = true,
SchemaName = "appName"
},
new[] { HangfireHelper.QueueName }));

services.AddHangfireServer(options =>
{
options.Queues = new[] { HangfireHelper.QueueName };
});

```

QueueName is "queue-{machine name}"

And jobs are added like this

```
var queueName = HangfireHelper.QueueName;
JobStorage.Current = new MAMQSqlServerStorage(Configuration.GetConnectionString("HangfireConnection"), new SqlServerStorageOptions()
{
SchemaName = HangfireHelper.Schema
}, new[] { queueName });

using (var connection = JobStorage.Current.GetConnection())
{
foreach (var recurringJob in connection.GetRecurringJobs())
{
if (recurringJob.Queue == queueName)
{
RecurringJob.RemoveIfExists(recurringJob.Id);
}
}
}

/////
RecurringJob.AddOrUpdate("JobName", () => jobService.DoJobAsync(param), appSettings.HangfireJobs.JobName, queue: queueName);

```

To me it looks like the newest instance becomes a "main" instance and when it gets shut down, the existing instance doesn't take over.
My desired outcome is if there are multiple servers the job should execute only on one server, not all of them (doesn't matter which one) and when I shut down some instance it shouldn't stop the jobs from getting queued, other instances should still be able to pick them up.

Contributor guide

Open the contributing guide

Research direction

No files or tests are identified. Start by reproducing the shared-database setup with multiple servers, then inspect how recurring jobs are scheduled when one server stops; done means another running server queues the job and it executes only once.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, sql
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.