HangfireIO / HangfireIO/Hangfire
Jobs are picked up from queues that the server is not supposed to listen to
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
I found several issues that sound similar but never a real solution.
I have two Hangfire servers. Let's call them A and B. Server A is configured to listen to QueueA (and only QueueA, no "default" queue!) and Server B listens to QueueB (also no "default" queue). I have a job that is scheduled on QueueA (doesn't matter if I schedule it as a background or a recurring job). But the job is (sometimes) picked up by Server B.
```` csharp
services.AddHangfireServer((sp, options) =>
{
options.Queues = new[] { "queue_a" };
options.WorkerCount = 1;
options.ServerName = "server-a";
});
services.AddHangfireServer((sp, options) =>
{
options.Queues = new[] { "queue_b" };
options.WorkerCount = 1;
options.ServerName = "server-b";
});
recurringJobManager.AddOrUpdate(
nameof(JobA),
"queue_a",
job => job.Execute(CancellationToken.None),
"");
````
The problem: Server B does not reference the same set of assemblies and is thus not even able to load the job from the database. Instead of letting the other server pick it up, Server B keeps trying until the retry attempts are exhausted and the job fails.
```` csharp
Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details.
---> System.IO.FileNotFoundException: Could not resolve assembly 'JobA'.
at System.Reflection.TypeNameParser.ResolveAssembly(String assemblyName)
at System.Reflection.TypeNameParser.GetType(String typeName, ReadOnlySpan`1 nestedTypeNames, String assemblyNameIfAny)
at System.Reflection.TypeNameParser.NamespaceTypeName.ResolveType(TypeNameParser& parser, String containingAssemblyIfAny)
at System.Reflection.TypeNameParser.Parse()
at System.Reflection.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Assembly requestingAssembly, Boolean throwOnError, Boolean ignoreCase, Boolean extensibleParser)
at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName) in C:\projects\hangfire-525\src\Hangfire.Core\Common\TypeHelper.cs:line 78
at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 96
--- End of inner exception stack trace ---
at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 120
at Hangfire.RecurringJobExtensions.TriggerRecurringJob(IBackgroundJobFactory factory, JobStorage storage, IStorageConnection connection, IProfiler profiler, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\RecurringJobExtensions.cs:line 115
at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\Server\RecurringJobScheduler.cs:line 333
````
I have a job filter that makes sure the original queue is used, even on retries. So the issue is not related to the job being re-queued with the wrong queue.
Why does a server pick up jobs from queues, that it is not meant to listen to and how can i fix that?
BTW: This issue is not related to the storage used. I see it with SqlServer, Redis and PostgreSql.
Contributor guide
Research direction
Start by tracing queue selection and recurring-job scheduling through Hangfire.Server.RecurringJobScheduler.cs and Hangfire.RecurringJobExtensions.cs, then follow job deserialization in Hangfire.Storage.InvocationData.cs and Hangfire.Common.TypeHelper.cs. Use the two-server QueueA/QueueB reproduction as the check: Server B should not pick up or repeatedly retry jobs assigned to QueueA.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100