HangfireIO / HangfireIO/Hangfire

Migration .net 3.1 core -> .net 7 - makes hangfire hiccup

Open
#2,336 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

I migrated my solution from 3.1 to .NET 7. Everything seems to work perfectly until after a few hours (there is no exact time). Hangfire stops processing and starts queuing up to the available works limit (see image 3). As soon as it reaches the limit, it starts placing jobs in the queued state.
![crash_](https://github.com/HangfireIO/Hangfire/assets/5939581/b8a1ae16-d542-4037-9e17-449050ff075e)

Interestingly, from time to time, it ‘recovers itself’ itself, processing 5x more than normal. It typically processes between 30 and 80 jobs per second. However, when this issue occurs, it reaches peaks of 500 jobs. Unfortunately, there are times when it doesn’t recover, and the only solution is to restart the application (docker image) for it to return to normal.
![top_500](https://github.com/HangfireIO/Hangfire/assets/5939581/746e95df-2754-43fc-bdbf-f55b943be245)

I have to revert the software version to 3.1, unfortunately.

My code:
```
services.AddHangfireServer();
services.AddHangfire((provider, config) =>
{
provider.GetRequiredService().Configure(config);
});
services.AddHangfireServer((provider, options) =>
{
var config = provider.GetRequiredService();
options.ServerName = "Shared";
options.WorkerCount = config.WorkerCountModifierShared * config.WorkerCountMultiplierShared;
options.Queues = new string[] { "sh_operation_service", "sh_couchdb_service", "sh_couchdb_expurgo_service", "sh_couchdb_service_upload", "default" };
});

services.AddHangfireServer((provider, options) =>
{
var config = provider.GetRequiredService();
options.ServerName = "Cecom";
options.WorkerCount = config.WorkerCountModifierCecom * config.WorkerCountMultiplierCecom;
options.Queues = new string[] { "cecom_service", "talao_service", "default" };
});

services.AddHangfireServer((provider, options) =>
{
var config = provider.GetRequiredService();
options.ServerName = "HighPriority";
options.WorkerCount = config.WorkerCountModifierHighPriority * config.WorkerCountMultiplierHighPriority;
options.Queues = new string[] { "hp_couchdb_service", "hp_couchdb_expurgo_service", "default" };
});

services.AddHangfireServer((provider, options) =>
{
var config = provider.GetRequiredService();
options.ServerName = "LowPriority";
options.WorkerCount = config.WorkerCountModifierLowPriority * config.WorkerCountMultiplierLowPriority;
options.Queues = new string[] { "lp_couchdb_service", "lp_couchdb_expurgo_service", "default" };
});

services.AddHangfireServer((provider, options) =>
{
var config = provider.GetRequiredService();
options.ServerName = "HighPriorityNoParallel";
options.WorkerCount = 1;
options.Queues = new string[] { "hp_couchdb_service_dwp" };
});

services.AddHangfireServer((provider, options) =>
{
var config = provider.GetRequiredService();
options.ServerName = "BackgroudOffice";
options.WorkerCount = config.WorkerCountModifierBackground * config.WorkerCountMultiplierBackground;
options.Queues = new string[] { "bg_couchdb_service" };
});
```
```
var bgJob = app.ApplicationServices.GetRequiredService();
var storage = bgJob.Storage;

app.UseHangfireDashboard("/hangfire", new DashboardOptions
{
DisplayStorageConnectionString = false,
Authorization = Enumerable.Empty(),
}, storage: storage);

ccRecurrent = new RecurringJobManager(storage);
```

in the log system (SEQ + Serilog) no error or warning that shows the reason, nothing different.

SQL Version 2016
.NET 7.0.14
Hangfire 1.8.6
image docker: 7.0-jammy (ubuntu)
VM OS: Ubuntu 20.04.1 LTS
PackageReference Include="Hangfire" Version="1.8.6" //only this package

![glances](https://github.com/HangfireIO/Hangfire/assets/5939581/4909cae6-ee55-4d9c-86c8-431e51570593)
_the server is very big, F16 on Azure (16 vcpu and 32ram) according to htop and looks I don’t even use 20% of it_

![normal](https://github.com/HangfireIO/Hangfire/assets/5939581/ca203a7e-be37-4593-b90e-b2fad169ef5a)

![db3ca0df-6364-4c6d-96df-cd7edfbdf8bf](https://github.com/HangfireIO/Hangfire/assets/5939581/f3756331-2393-46af-bbb1-c2a6bacbd535)

Contributor guide

Open the contributing guide

Research direction

Start with the provided AddHangfireServer configuration and the Hangfire 1.8.6 setup, then investigate processing and storage behavior on .NET 7 with SQL Server 2016. Compare logs and queue/worker metrics during normal processing and after the intermittent stall. Done means identifying a reproducible cause and restoring stable job processing without restarting the Docker application.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.