HangfireIO / HangfireIO/Hangfire

Jobs are picked up from queues that the server is not supposed to listen to

Open
#2,401 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

I found several issues that sound similar but never a real solution.

I have two Hangfire servers. Let's call them A and B. Server A is configured to listen to QueueA (and only QueueA, no "default" queue!) and Server B listens to QueueB (also no "default" queue). I have a job that is scheduled on QueueA (doesn't matter if I schedule it as a background or a recurring job). But the job is (sometimes) picked up by Server B.

```` csharp
services.AddHangfireServer((sp, options) =>
{
options.Queues = new[] { "queue_a" };
options.WorkerCount = 1;
options.ServerName = "server-a";
});

services.AddHangfireServer((sp, options) =>
{
options.Queues = new[] { "queue_b" };
options.WorkerCount = 1;
options.ServerName = "server-b";
});

recurringJobManager.AddOrUpdate(
nameof(JobA),
"queue_a",
job => job.Execute(CancellationToken.None),
"");
````

The problem: Server B does not reference the same set of assemblies and is thus not even able to load the job from the database. Instead of letting the other server pick it up, Server B keeps trying until the retry attempts are exhausted and the job fails.

```` csharp
Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details.
---> System.IO.FileNotFoundException: Could not resolve assembly 'JobA'.
at System.Reflection.TypeNameParser.ResolveAssembly(String assemblyName)
at System.Reflection.TypeNameParser.GetType(String typeName, ReadOnlySpan`1 nestedTypeNames, String assemblyNameIfAny)
at System.Reflection.TypeNameParser.NamespaceTypeName.ResolveType(TypeNameParser& parser, String containingAssemblyIfAny)
at System.Reflection.TypeNameParser.Parse()
at System.Reflection.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Assembly requestingAssembly, Boolean throwOnError, Boolean ignoreCase, Boolean extensibleParser)
at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName) in C:\projects\hangfire-525\src\Hangfire.Core\Common\TypeHelper.cs:line 78
at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 96
--- End of inner exception stack trace ---
at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 120
at Hangfire.RecurringJobExtensions.TriggerRecurringJob(IBackgroundJobFactory factory, JobStorage storage, IStorageConnection connection, IProfiler profiler, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\RecurringJobExtensions.cs:line 115
at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\Server\RecurringJobScheduler.cs:line 333
````

I have a job filter that makes sure the original queue is used, even on retries. So the issue is not related to the job being re-queued with the wrong queue.

Why does a server pick up jobs from queues, that it is not meant to listen to and how can i fix that?

BTW: This issue is not related to the storage used. I see it with SqlServer, Redis and PostgreSql.

Contributor guide

Open the contributing guide

Research direction

Start by tracing queue selection and recurring-job scheduling through Hangfire.Server.RecurringJobScheduler.cs and Hangfire.RecurringJobExtensions.cs, then follow job deserialization in Hangfire.Storage.InvocationData.cs and Hangfire.Common.TypeHelper.cs. Use the two-server QueueA/QueueB reproduction as the check: Server B should not pick up or repeatedly retry jobs assigned to QueueA.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.