HangfireIO / HangfireIO/Hangfire

Preserve job queue over retries

Open
#1,695 2 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

Hi I could not find any issues on this raised so here goes.

using Hangfire 1.7.11 and Hangfire.PostgreSql 1.7.0

I have many different servers serving many different queues. The individual servers are not meant to be able to handle other queues than specified so as to balance the size and jobs of servers.
It would be great to have a built in option to preserve the queue when retrying a job.

For example I have many jobs related to many web api's on different queues.
When one of these web api's have a transient failure while a job is running and the job rightfully fails it is not useful for the retry to be scheduled on the default queue as it is then a toss up whether or not the next taker can even activate the dependencies for that job.

This means that instead of the intended retry the system has to wait for the next scheduled attempt on the right queue to have a realistic chance at succeeding said job, this leads to inconsistencies in job failure handling proportionally to the amount of queues.

Sandeepk87 proposes a solution based on a jobfilter attribute here:
https://discuss.hangfire.io/t/rescheduling-the-job-to-default-queue-despite-it-is-being-set-in-another-queue/1175/11

I have made a single minor modification to it and commented as well on the thread.
This is the resultant attribute source code:

using Hangfire.Common;
using Hangfire.States;
using Hangfire.Storage;

namespace XXXXX.DbContexts.Attributes
{
    public class PreserveOriginalQueueAttribute : JobFilterAttribute, IApplyStateFilter, IElectStateFilter
    {
        public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
        {
            var id = context.BackgroundJob.Id;
            var state = context.Connection.GetStateData(id);
            var initialQueue = context.Connection.GetJobParameter(id, JobParam.InitialQueue);
            state.Data.TryGetValue("Queue", out string name);
            if (string.IsNullOrEmpty(initialQueue))
            {
                context.Connection.SetJobParameter(id, JobParam.InitialQueue, name);
            }
            if (state.Name.Equals(States.Scheduled) && state.Reason.StartsWith(Reason.Retry))
            {
                //FailedState
                if (context.NewState is EnqueuedState newState)
                {
                    newState.Queue = initialQueue;
                    context.Connection.SetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason, Reason.Retry);
                }
            }
            if (context.NewState is ProcessingState processingState)
            {
                //Remove QueueReason Param for subsequent manual triggers.

                context.Connection.SetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason, null);
            }
        }
        public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
        {
        }

        /// For changing state from default to original queue for manual trigger

        public void OnStateElection(ElectStateContext context)
        {
            var initialQueue = context.Connection.GetJobParameter(context.BackgroundJob.Id, JobParam.InitialQueue);
            var queueReason = context.Connection.GetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason);
            if (context.CurrentState.Equals(States.Enqueued) && !string.IsNullOrEmpty(initialQueue) && string.IsNullOrEmpty(queueReason))
            {
                context.Connection.SetJobParameter(context.BackgroundJob.Id, JobParam.QueueReason, Reason.Trigger);
                context.CandidateState = new EnqueuedState(initialQueue);
            }
        }
    }
    static internal class States
    {
        public static readonly string Enqueued = "Enqueued";
        public static readonly string Scheduled = "Scheduled";
    }
    static internal class JobParam
    {
        public static readonly string InitialQueue = "InitialQueue";
        public static readonly string QueueReason = "QueueReason";
    }
    static internal class Reason
    {
        public static readonly string Trigger = "Trigger";
        public static readonly string Retry = "Retry";
    }
}

Please consider adding an officially supported method of preserving the queue over retries as it is hard to understand all of the complexities and foresee if this filter will work in all cases and continue to work as intended throughout updates.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the supplied PreserveOriginalQueueAttribute and Hangfire's IApplyStateFilter and IElectStateFilter behavior during retries and manual triggers. Determine the supported design for retaining the original queue across state changes. Done means an officially supported method preserves queue affinity over retries without relying on an application-specific filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.