Background job being enqueued by wrong microservices

Hello,

Been looking at this for days, tried different approaches, but I can't seem to find the correct solution.
Situation:
- 2 microservices, each having their queues, MS 1 has queue A and MS 2 has queue B
- A job is being scheduled by MS 1 to be picked up by queue A.
- Sometimes MS 1 picks up this job, sometimes MS 2 tries to pick this up. If we're lucky, it succeeds picking it up on the correct MS, within 10 retries. If we're unlucky, MS 2 tries to pick it up.
- MS 1 and MS 2 have their own assemblies, own jobs, and own queues

So MS 2 does not know the code of the job being scheduled on queue A, and should not need to.
Typical error thrown from MS 2:
`System.IO.FileNotFoundException: Could not resolve assembly 'Integration.Domain'.
at System.TypeNameParser.ResolveAssembly(String asmName, Func`2 assemblyResolver, Boolean throwOnError, StackCrawlMark& stackMark)
at System.TypeNameParser.ConstructType(Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName)
at Hangfire.Storage.InvocationData.DeserializeJob()`

We have this attribute above the job:
` [PreserveOriginalQueue(Order = 1, Queue = "integration")]`

The attribute filter (taken from another contributor, slightly adjusted):
`public class PreserveOriginalQueueAttribute : JobFilterAttribute, IApplyStateFilter
{
public string Queue { get; set; }
public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
if(context.NewState is ScheduledState ss)
{
var originalQueue = SerializationHelper.Deserialize(context.Connection.GetJobParameter(context.BackgroundJob.Id, "Queue"));

if(originalQueue == null)
{
if (string.IsNullOrEmpty(Queue))
return;

originalQueue = Queue;
}

context.Connection.SetJobParameter(
context.BackgroundJob.Id,
"OriginalQueue",
SerializationHelper.Serialize(originalQueue));

}
else if (context.NewState is EnqueuedState es)
{
// Checking if an original queue is already set
var originalQueue = SerializationHelper.Deserialize(context.Connection.GetJobParameter(context.BackgroundJob.Id, "OriginalQueue"));

if (originalQueue == null)
{
if (!string.IsNullOrEmpty(Queue))
originalQueue = Queue;

if (string.IsNullOrEmpty(originalQueue))
originalQueue = es.Queue;
}

if (originalQueue != null)
{
// Override any other queue value that is currently set (by other filters, for example)
es.Queue = originalQueue;
}

// Queueing for the first time, we should set the original queue
context.Connection.SetJobParameter(
context.BackgroundJob.Id,
"OriginalQueue",
SerializationHelper.Serialize(es.Queue));

}
}

public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
return;
}
}`

Startup code for both MS 1 and MS 2:
` public static IServiceCollection AddSchedulerInstance(this IServiceCollection services, string hangfireSqlConnectionString, string[] queues, int workerCount = 10)
{
services.AddSchedulerClient(hangfireSqlConnectionString);

services.AddHangfireServer(opt =>
{
opt.Queues = queues;
opt.WorkerCount = workerCount;
});

return services;
}

public static IServiceCollection AddSchedulerClient(this IServiceCollection services, string hangfireSqlConnectionString)
{
services.AddHangfire((sp, cfg) =>
{
cfg.UseColouredConsoleLogProvider();
cfg.UseSimpleAssemblyNameTypeSerializer();
cfg.UseRecommendedSerializerSettings();
cfg.UseConsole(new ConsoleOptions()
{
FollowJobRetentionPolicy = true
});
cfg.UseSqlServerStorage(hangfireSqlConnectionString);
cfg.UseFilter(new PreserveOriginalQueueAttribute());
});
return services;
}`

Hope we can get this sorted!

Start with AddSchedulerInstance and AddSchedulerClient, then inspect PreserveOriginalQueueAttribute and the Hangfire storage configuration shown in the issue. Reproduce the two-service setup with separate queue lists and trace how the job is deserialized and selected. Done means jobs from queue A are not attempted by the service that only listens to queue B, without assembly-resolution failures.

Written by the indexing model from the issue text.