HangfireIO / HangfireIO/Hangfire
Preserve job queue over retries
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
Hi I could not find any issues on this raised so here goes.
using Hangfire 1.7.11 and Hangfire.PostgreSql 1.7.0
I have many different servers serving many different queues. The individual servers are not meant to be able to handle other queues than specified so as to balance the size and jobs of servers.
It would be great to have a built in option to preserve the queue when retrying a job.
For example I have many jobs related to many web api's on different queues.
When one of these web api's have a transient failure while a job is running and the job rightfully fails it is not useful for the retry to be scheduled on the default queue as it is then a toss up whether or not the next taker can even activate the dependencies for that job.
This means that instead of the intended retry the system has to wait for the next scheduled attempt on the right queue to have a realistic chance at succeeding said job, this leads to inconsistencies in job failure handling proportionally to the amount of queues.
Sandeepk87 proposes a solution based on a jobfilter attribute here:
https://discuss.hangfire.io/t/rescheduling-the-job-to-default-queue-despite-it-is-being-set-in-another-queue/1175/11
I have made a single minor modification to it and commented as well on the thread.
This is the resultant attribute source code:
using Hangfire.Common;
using Hangfire.States;
using Hangfire.Storage;
namespace XXXXX.DbContexts.Attributes
{
public class PreserveOriginalQueueAttribute : JobFilterAttribute, IApplyStateFilter, IElectStateFilter
{
public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
var id = context.BackgroundJob.Id;
var state = context.Connection.GetStateData(id);
var initialQueue = context.Connection.GetJobParameter(id, JobParam.InitialQueue);
state.Data.TryGetValue("Queue", out string name);
if (string.IsNullOrEmpty(initialQueue))
{
context.Connection.SetJobParameter(id, JobParam.InitialQueue, name);
}
if (state.Name.Equals(States.Scheduled) && state.Reason.StartsWith(Reason.Retry))
{
//FailedState
if (context.NewState is EnqueuedState newState)
{
newState.Queue = initialQueue;
context.Connection.SetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason, Reason.Retry);
}
}
if (context.NewState is ProcessingState processingState)
{
//Remove QueueReason Param for subsequent manual triggers.
context.Connection.SetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason, null);
}
}
public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
}
/// For changing state from default to original queue for manual trigger
public void OnStateElection(ElectStateContext context)
{
var initialQueue = context.Connection.GetJobParameter(context.BackgroundJob.Id, JobParam.InitialQueue);
var queueReason = context.Connection.GetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason);
if (context.CurrentState.Equals(States.Enqueued) && !string.IsNullOrEmpty(initialQueue) && string.IsNullOrEmpty(queueReason))
{
context.Connection.SetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason, Reason.Trigger);
context.CandidateState = new EnqueuedState(initialQueue);
}
}
}
static internal class States
{
public static readonly string Enqueued = "Enqueued";
public static readonly string Scheduled = "Scheduled";
}
static internal class JobParam
{
public static readonly string InitialQueue = "InitialQueue";
public static readonly string QueueReason = "QueueReason";
}
static internal class Reason
{
public static readonly string Trigger = "Trigger";
public static readonly string Retry = "Retry";
}
}
Please consider adding an officially supported method of preserving the queue over retries as it is hard to understand all of the complexities and foresee if this filter will work in all cases and continue to work as intended throughout updates.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the supplied PreserveOriginalQueueAttribute and Hangfire's IApplyStateFilter and IElectStateFilter behavior during retries and manual triggers. Determine the supported design for retaining the original queue across state changes. Done means an officially supported method preserves queue affinity over retries without relying on an application-specific filter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100