HangfireIO / HangfireIO/Hangfire
Moving scheduled jobs to enqueued state when assembly may be missing
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
We've been using the feature of queues to separate jobs between processing servers using a common Hangfire SQL instnace. This all works fine as long as jobs are scheduled on the client/producer, as jobs are moved to the Enqueued state with the appropriate queue when the job is scheduled.
However whenever we schedule a job with a delay, the job moves to the Scheduled state until the job scheduler process on our processing service detects it is ready and moves it to the Enqueued state. As I understand it, this is also essentially how the HF automatic retry on failed jobs - it moves the job back to the scheduled state.
This is a problem when the processing service does not have the target assembly of the job class. It cannot move the job to the Enqueued state because I'm guessing the scheduler uses reflection to resolve the job's assembly. This results in an unhandled exception and the Hangfire retry mechanism kicks in, but it will just repeat this failed strategy until the number of allowed attempts is exceeded and the job is moved to the failed state. While it is possible a service that does have the assembly needed will get the chance to move the job to the Enqueued state, the job could remain in auto-retry limbo for an indeterminate amount of time, or as we found, the auto-retry limit is reached very quickly, especially with numerous services processing jobs.
Is it possible to modify the job scheduler to respect the queue assigned to a job and not attempt to enqueue the job? Or, can we get the job scheduler to enqueue a scheduled job without an assembly reference?
Contributor guide
Research direction
Start with the job scheduler path that moves delayed or retried jobs from Scheduled to Enqueued, focusing on queue assignment and target-assembly resolution. Reproduce the case where the processing service lacks the job assembly; done means the scheduled job can reach Enqueued using its assigned queue without failing assembly resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100